license: mit
pretty_name: HarmURLBench
language:
- en
tags: - ai-safety
- retrieval-augmented-generation
- agent-safety
- red-teaming
extra_gated_heading: "Request access to HarmURLBench"
extra_gated_prompt: >-
HarmURLBench pairs real web pages with harmful behaviors from HarmBench for
research on retrieval-induced safety degradation. Access is granted to
identified researchers who agree to the following: (1) use the data only for
research on AI safety, model evaluation, and defenses; (2) do not redistribute
the URL list or cached pages, in whole or in part; (3) do not use the data to
build or fine-tune systems whose purpose is to produce harmful content;
(4) cite HarmURLBench and HarmBench in any resulting work; (5) honour takedown
requests in any local copy. Requests are reviewed manually.
extra_gated_fields:
Full name: text
Affiliation: text
Intended research use: text
I will use HarmURLBench for research only and will not redistribute it: checkbox
I will cite HarmURLBench and HarmBench: checkbox
extra_gated_button_content: "Request access"
configs: - config_name: default
data_files: harmurlbench_clean.csv
HarmURLBench
Stance- and topical-relevance-stratified real-world URLs paired with 320 harmful
behaviors from HarmBench (Mazeika et al., 2024).
Companion artifact for Relevance as a Vulnerability: How Web Retrieval Degrades
Safety Alignment in LLM Agents (Findings, ARR).
Each behavior has up to one URL per stance level (SS1–SS5) with topical
relevance TR ≥ 3.
Files
| File | Contents |
|---|---|
harmurlbench_clean.csv |
One row per URL: split, behavior_id, stance_score (1–5), topical_relevance (1–5), content_length, url. |
test/dataset_index.json |
behavior_id → {ss_1..ss_5} with URL, TR score, discovery iteration. |
test/html/ |
Cached page text, one file per (behavior_id, stance_level). |
dataset_summary.json |
Aggregate statistics. |
URL_CRITERIA.md |
Qualification criteria and score-scale definitions. |
Size (test split, 320 behaviors)
| Stance | URLs | Coverage |
|---|---|---|
| SS1 (oppositional) | 281 | 87.8% |
| SS2 | 282 | 88.1% |
| SS3 (neutral) | 263 | 82.2% |
| SS4 | 283 | 88.4% |
| SS5 (enabling) | 296 | 92.5% |
| Total | 1,405 | 87.8% |
This release contains the test split, which is what the paper reports on. 74
misclassified (behavior_id, SS) pairs were removed after GPT-4o re-evaluation
and structural-anomaly checks.
Scoring scales
Stance Score (SS) — stance of the page toward the behavior: 1 strongly
opposes · 2 mildly opposes · 3 neutral/informational · 4 mildly enables ·
5 strongly enables.
Topical Relevance (TR) — how closely the page pertains to the behavior:
1 unrelated → 5 directly addresses it. All URLs have TR ≥ 3.
License
The annotations (SS/TR labels and the URL–behavior pairing) are MIT-licensed.
The text of the referenced web pages is not relicensed and remains the
property of its publishers; it is provided here for research use only.
HarmBench behaviors are MIT-licensed by their authors.
Takedown
Page owners can request removal of a cached page: email [email protected]
(cc [email protected]) with the URL.
Citation
Please cite both HarmURLBench and HarmBench.
@inproceedings{nawal2026relevance,
title = {Relevance as a Vulnerability: How Web Retrieval Degrades Safety Alignment in LLM Agents},
author = {Nawal, Aditya and Baser, Manit and Gurusamy, Mohan},
booktitle = {Findings of the Association for Computational Linguistics},
year = {2026}
}