license: other
task_categories:
- text-generation
tags: - ai-safety
- refusal
- over-refusal
- red-teaming
- activation-steering
Over-refusal evaluation data
The frozen prompt splits behind the over-refusal experiments. One draw is shared by every
model — Qwen2.5-3B, Llama-3.1-8B and Gemma-2-9B all read the same data.json, verified by
sha256 — so cross-model comparisons are paired on the same prompts rather than being two
independent samples.
| split | rows | sources | role |
|---|---|---|---|
| train | 512 | OR-Bench-80k (benign) | LoRA probe training; each prompt is paired with a fixed refusal target |
| extract | 128 | OR-Bench-80k (benign) | prompts the NAS direction is read from; disjoint from train |
| validation | 512 | OR-Bench-hard 256 + OR-Bench-toxic 256 | layer/alpha selection |
| test | 1750 | OR-Bench-hard 1000 + XSTest 450 + HarmBench 300 | held out, touched once |
| capability | 128 | MMLU validation sample | capability guard, explicitly not full MMLU |
| toxic_followup | 399 | OR-Bench-toxic | prompts no split used; frozen configurations, no retuning |
Seed 42. toxic_followup is frozen separately from data.json and is disjoint from every split
in it.
Files
data.json— the artifact the pipeline reads, byte-identical toexperiments/reproduce_overrefusal/runs/baseline/data.json. Holdssplits,audit,provenance,deduplicationandtest_harmbench_scope.toxic_followup.json— the follow-up set in the same shape.splits/*.jsonl— the same rows, one JSON object per line, for loaders.manifest.json— sha256 of everything, per-split source breakdown, and the upstream
provenance the run captured: Hugging Face dataset fingerprints and the HarmBench CSV hash.
Every row carries id, prompt, source, kind (benign, harmful or capability) andcategory. train rows add the canned response used as the probe's target; capability rows
add answer.
from datasets import load_dataset
test = load_dataset("json", data_files="splits/test.jsonl", split="train")
Why one shared draw matters
The over-refusal protocol selects a layer and an alpha on validation and then reports on test.
If each model drew its own prompts, a difference between models would mix the steering effect with
sampling noise from two different prompt sets. Sharing one draw removes that: the models differ,
the prompts do not. reuse_data_from in each model's config points at the run that made the draw,
and the pipeline refuses to copy unless the data block and the seed match.
Integrity
id is a fingerprint of the prompt, and the pipeline recomputes the audit on every read: if
any id disagrees with its prompt, or one question appears in two splits, the run stops. That check
is what makes "held out" mean something here.
Deduplication is NFKC + casefold + whitespace normalisation; semantic paraphrase overlap between
splits is not guaranteed. The HarmBench portion covers the standard and contextual functional
categories only.
Licensing
These are prompts collected from upstream datasets — OR-Bench, XSTest, HarmBench and MMLU — each
under its own licence and terms. This folder redistributes them for reproducibility of the
experiments; check the upstream terms before using them for anything else. The HarmBench rows are
harmful requests by construction and exist to measure a safety regression, not to be executed.