license: cc-by-nc-4.0
language:

• en
  pretty_name: PatchBench
  tags:
• language-model-safety
• red-teaming
• refusal
• robustness
• benchmark
• harmful-prompts
• prompt-paraphrasing

PatchBench

PatchBench accompanies a study of local safety patching for instruction-tuned language models. The benchmark evaluates whether a patching method fixes harmful prompts that a model answered unsafely, while also measuring whether the method preserves behavior on nearby benign prompts.

The dataset contains model-specific prompt pools, target-model answers, safety labels, and grouped validation/test examples built around harmful source prompts and local prompt variants.

The released benchmark covers eight model-specific folders:

gemma-3-12b-it
gemma-3-4b-it
llama-3.1-8B-Instruct
llama4_scout-it
ministral-3-14B-Instruct-2512
mistral-7B-Instruct-v0.3
qwen2.5-14B-Instruct
qwen2.5-3B-Instruct

Dataset Structure

The main benchmark files are under:

final_dataset/

Each model-specific directory contains:

row_data.json
validation.json
test.json

row_data.json is a flat model-specific inference and labeling pool. Each row has the following fields:

id
prompt
answer
label

The prompt field contains the input prompt, answer contains the corresponding target-model answer, and label contains the safety label assigned to that answer.

validation.json contains grouped validation examples for the corresponding model. Across all model folders, the release contains 160 validation examples.

test.json contains grouped test examples for the corresponding model. Across all model folders, the release contains 402 test examples.

Each validation/test example has the following top-level fields:

id
original_prompt
groups

The groups object contains:

close_harmful
benign_same_structure
benign_harmful_words

For each example, close_harmful contains:

paraphrases
translations
punctuation
case_inversion
spelling_mistakes

In the released validation and test files, each example contains 10 paraphrases, 2 translations, 3 punctuation variants, 1 case-inversion variant, 3 spelling-mistake variants, 5 benign same-structure prompts, and 5 benign harmful-word prompts.

Additional construction artifacts are under:

alpaca/
data_processing/

alpaca/alpaca_for_condition.json contains Alpaca-derived safe prompts.

alpaca/alpaca_for_refusal.json contains Alpaca-derived prompt records split into train and test.

alpaca/behavior_refusal.json contains compliant and non-compliant response suffixes.

data_processing/row_data/redbench_filtered.json contains filtered RED-Bench source prompts.

data_processing/model_inference/ contains model inference outputs with prompt and answer fields.

data_processing/wildguard_inference/ contains WildGuard labeling outputs.

data_processing/elo_ranking/ contains Elo-ranking artifacts.

data_processing/selected-70-harmful/ contains selected harmful examples for each target model.

Croissant metadata is available at:

croissant.json

Intended Use

PatchBench is intended for research on language-model safety, including evaluation of safety patching, refusal behavior, robustness to local prompt variants, and preservation of benign behavior near harmful prompts.

PatchBench is not intended for training models to comply with harmful requests or for generating actionable harmful content.

Ethics Statement

PatchBench contains harmful, unsafe, and adversarial prompt text, as well as model answers to harmful prompts. It is released to support controlled safety research and reproducible benchmarking.

Users should handle examples carefully, especially when displaying, sharing, or analyzing prompt and answer content. Users should not use PatchBench to train models to comply with harmful requests or to generate actionable harmful content.

The dataset files contain prompt text, generated prompt variants, model answers, labels, and construction metadata. They are not structured as personal records about individuals.

Licenses

The source data used to construct PatchBench are released under permissive or research-compatible licenses. RED-Bench prompts are released under the MIT License. Alpaca-derived data are based on Stanford Alpaca, which is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license.

PatchBench also includes target-model outputs generated during benchmark construction. These outputs were produced using models released under their respective licenses or terms, including Apache-2.0–licensed Qwen and Mistral models, Gemma models under the Gemma Terms of Use, and Llama models under the Llama Community License.

PatchBench itself is released under CC BY-NC 4.0 to remain consistent with the licenses and usage restrictions of the upstream materials.

Users should provide appropriate attribution to PatchBench, RED-Bench, Stanford Alpaca, and the relevant model providers, and should not imply endorsement by any upstream provider.

Code

The accompanying code is available at:

https://github.com/ano6161/anonymous-patchbench

Data Sources

• RED-Bench: https://huggingface.co/datasets/knoveleng/redbench
• Stanford Alpaca: https://github.com/tatsu-lab/stanford_alpaca

Used Models

• Qwen2.5-3B-Instruct: https://huggingface.co/Qwen/Qwen2.5-3B-Instruct
• Qwen2.5-14B-Instruct: https://huggingface.co/Qwen/Qwen2.5-14B-Instruct
• Gemma-3-4B-IT: https://huggingface.co/google/gemma-3-4b-it
• Gemma-3-12B-IT: https://huggingface.co/google/gemma-3-12b-it
• Mistral-7B-Instruct-v0.3: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3
• Ministral-3-14B-Instruct-2512: https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512
• Llama-3.1-8B-Instruct: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
• Llama-4-Scout-17B-16E-Instruct: https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct