license: cc-by-4.0
language:

• en
  task_categories:
• text-classification
  tags:
• ai-safety
• jailbreak
• red-teaming
• llm-as-judge
• evaluation
• calibration
• adversarial-robustness
• harmbench
• arxiv:2606.25487
  pretty_name: "JudgeFlip: Jailbreak Judge Calibration & Robustness Audit"
  size_categories:
• n<1K

JudgeFlip — Jailbreak Judge Calibration & Adversarial-Robustness Audit

Companion metadata & reproduction dataset for the paper:

How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring
Yang Gao (Veyon Solutions). arXiv:2606.25487
Code: https://github.com/gy15901580825/judgeflip

Almost every LLM jailbreak / prompt-injection paper reports an attack-success rate (ASR), and
that number is assigned by an automated judge — a fine-tuned safety classifier, or an LLM
prompted to grade. This dataset lets you audit those judges: how well they agree with human labels
(calibration), and how easily their verdicts flip under content-preserving attacks (robustness).

⚠️ What this dataset is (and is not)

This is a responsible, metadata-only release. It deliberately contains no harmful text.

• ✅ It does contain: our judges' per-item verdicts, harm probabilities, human gold labels
  (majority of 3), attack-flip outcomes, aggregate calibration/robustness metrics, and the full
  human-annotation protocol.
• ❌ It does not contain: the harmful behaviors, the model completions being judged, or the
  attacked ("flipped") responses. Those live in HarmBench
  under HarmBench's own license, and are not redistributed here.

Every per-item row is keyed by HarmBench's behavior_id, so anyone with legitimate HarmBench access
can join back to the original text and fully reproduce our results — without this repository ever
hosting harmful content. This mirrors the withholding decision stated in the paper and the code repo.

Files

Note on per_item_metadata.csv: this file is produced from the paper's GPU experiment outputs
(step2_results.csv, flipped_pairs.csv) by scripts/build_metadata.py, which keeps only IDs,
predictions and labels and drops every free-text column (behavior, context,
original_response, attacked_response, generation). See the schema below.

per_item_metadata.csv schema

human_audit_round2.csv schema

The paper's harm-preservation audit (R4): 80 randomly-sampled flipped (judge Yes→No) attacked
responses, labeled by 2 independent annotators, blind, after a calibration gate. This file
contains the labels only — the underlying harmful text is withheld (join on item_id requires the
withheld master).

Raw 2-annotator agreement = 93.8% (75/80); all 5 disagreements were adjudicated to confirmed
harmful → human-verified harm-preservation = 100% (80/80), corroborating that the attacks never
alter the harmful body.

Headline findings (from the paper)

Safety judges are miscalibrated in opposite directions, and differ enormously in robustness:

• The dedicated classifier over-flags (recall 0.97) → inflates ASR before any attack.
• LLM-as-judges have high precision but wildly inconsistent recall (0.06–0.65) → ASR is both
  biased and unstable across judge choice.
• A prepended refusal sentence flips 39–88% of LLM-judge true positives — they key on refusal
  phrasing, not content.
• Even the robust classifier is broken 70% (21/30) of the time by a white-box GCG attack.
• A 2-annotator human audit confirms the attacks preserve harmful content (100% of 80 sampled flips).

Takeaway: no judge family should be trusted without calibrating and adversarially checking it.
When you report ASR, also report the judge's precision/recall and an adversarial-robustness check.

How to reproduce

1. Obtain the HarmBench classifier validation set + behaviors from
   HarmBench (subject to their license).
2. Join per_item_metadata.csv on behavior_id to recover the text.
3. Run the experiment scripts in the JudgeFlip repo
   (step2–step9), pinned to transformers==4.46.3.

License

The contents of this repository (our metrics, per-item verdicts/labels, annotation protocol,
and scripts) are released under CC-BY-4.0 — please cite the paper if you use them.

This repository does not include HarmBench data or model weights. HarmBench behaviors and
validation set are governed by HarmBench's own license and terms;
the evaluated models are under their respective licenses.

Citation

@misc{gao2026judgeflip,
  title         = {How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring},
  author        = {Gao, Yang},
  year          = {2026},
  eprint        = {2606.25487},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL}
}