license: mit
task_categories:

• text-generation
  language:
• en
  tags:
• abliteration
• conditioning
• safety-evaluation
  size_categories:
• n<1K
  configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*
  • split: test
    path: data/test-*

harmful-prompts-bench

Harmful prompts from AdvBench, the harmful half of a conditioning pair used to locate a refusal direction during abliteration.

What it is for

The repo holds English harmful prompts in a single text column, as ZStandard parquet with a train and a test split.
Abliteration measures the difference between a model's activations on harmless and harmful prompts, and this is the harmful side.
It is a measurement instrument, not training data. No model is fitted to these rows.
The train split locates the refusal direction and the test split measures refusals afterwards.
The harmless half is chibifire/harmless-prompts-en-train, and the code lives in weftspun/request-for-discussion.

Provenance

• Upstream source: AdvBench, from Zou et al., "Universal and Transferable Adversarial Attacks on Aligned Language Models". Its licence is MIT, from the llm-attacks repository.
• Obtained via mlabonne/harmful_behaviors at revision 01cead01398926d81f7c52bdb790ee8cf77ebba7. That repo repackages AdvBench and states no licence of its own.
• Processing: train and test concatenated, deduplicated on exact text, sorted lexicographically. Sorting makes the order deterministic, so there is no seed.
• A fifth of the rows is held out for test. Assignment is blake2b(text) % 5 == 0.
• Citation metadata is in CITATION.cff.

Use

heretic --model <model> \
  --good-prompts.dataset chibifire/harmless-prompts-en-train \
  --good-prompts.column text \
  --bad-prompts.dataset chibifire/harmful-prompts-bench \
  --bad-prompts.column text

from datasets import load_dataset
ds = load_dataset("chibifire/harmful-prompts-bench")

Licence

MIT, inherited from AdvBench.