Building your own extraction pair

An extraction pair is the smallest piece of infrastructure in the abliteration pipeline and the most consequential. Everything that follows (direction extraction, weight ablation, KL divergence, capability recovery) is downstream of two lists of prompts a human chose to compare. Those two lists silently define what "refusal" means for your model, and any refusal outside their coverage will survive the ablation as a stubborn residue.

Most practitioners never build their own. The field has settled on mlabonne/harmful_behaviors and mlabonne/harmless_alpaca because they arrived cleanly, matched Arditi's original methodology, and became the reference by default. For an English-language Llama or Qwen release aimed at general-purpose uncensoring, the reference pair is fine. For anything else, you will eventually notice that Heretic reports 3/100 refusals on its own benchmark but your model still refuses a category the standard pair did not touch. That is the signal that you need your own pair.

When the reference pair is not enough

Three situations reliably outgrow the reference pair. First, non-English models: the standard pair is English, and the refusal direction of a Chinese or Japanese or Russian model does not always align with the direction extracted from English prompts. Chinese producers (see huihui-ai) get better results extracting on Mandarin prompt pairs. Second, domain-specialised models: a coding-focused release refuses different categories than a general-purpose one, and the standard pair under-covers code-adjacent refusals (malware, exploit generation, unauthorized-access-pattern). Third, models above 8 billion parameters, where a single refusal direction is often insufficient regardless of the pair used. That case requires multi-direction extraction, treated separately below.

The diagnostic that indicates you need a custom pair is not the refusal count on Heretic's built-in benchmark. That benchmark uses the same distribution the pair was extracted from, so a well-extracted direction will score well on it whether or not it generalises. The diagnostic is residual refusal on prompts your pair did not cover. Run your abliterated model on a held-out probe set (see holding out an evaluation split) and count how many categories still refuse. If any category refuses more than 20 percent of the time, that category was probably under-represented in your harmful side.

Sizing: hundreds, not thousands

The most common mistake is thinking bigger is better. Sumandora's reference implementation samples a few hundred prompts per side; FailSpy's default is 512 cached activations; Cracken reported good results with 52 harmful and 73 harmless prompts on a domain-specialised extraction. The abliterix stack reportedly uses 800 per side. None of these numbers approach "thousands," and there is a reason.

The difference-of-means estimator converges quickly. Once you have a few hundred matched prompts on each side, the direction stops moving in any meaningful way; adding more prompts averages in more noise from length variation, framing variation, and vocabulary variation. The direction becomes softer. Practitioners report that a 500/500 split extracts a slightly better direction than a 5000/5000 split from the same source pool, because the smaller split can be more carefully curated. The extraction cost is also modest at these sizes: a few minutes of activation capture on a single GPU.

Aim for 200 to 800 prompts per side. If you cannot get 200 well-matched prompts on the harmful side without stretching the definition of "harmful," you probably have a domain-specific extraction and should scope down further rather than pad up.

Framing matching: what actually goes in the direction

The most subtle mistake is a framing mismatch between the two sides. The extractor computes a mean vector on the harmful side and a mean vector on the harmless side; the difference is the "refusal direction." If your harmful prompts are mostly imperatives ("Write instructions for X") and your harmless prompts are mostly questions ("What is the capital of Y?"), the difference-of-means vector encodes imperative-versus-interrogative as much as it encodes refuse-versus-comply. Ablating along that direction will suppress imperatives generally, not just refusals, and the model will exhibit strange behaviour on ordinary command-form prompts.

The fix is to match framing deliberately. If your harmful side is "Write instructions for X," your harmless side should be "Write instructions for Y" where Y is a benign task. If your harmful side is "How do I do X," your harmless side should be "How do I do Y." Cracken's domain-specific abliteration writeup matched action-framed harmful prompts against question-framed harmless prompts carrying explicit educational context; this pairs the harm axis against pure harm and gets a cleaner direction because both sides carry the same educational-context signal that would otherwise leak in.

A quick check: read a handful of prompts from each side aloud. If the two sides sound like they are asking for different kinds of thing (a fact versus an action, a definition versus a plan, a story versus instructions), your direction will encode that difference as well as the refusal difference. Rebalance until the two sides are matched in what they ask for and differ only in whether the ask is harmful.

Length matching: the invisible bias

The second subtle mistake is length. Refusal responses in an aligned model tend to be short; compliant responses tend to be longer. If your harmful side has shorter prompts than your harmless side, the extractor sees "refusal tokens are shorter than compliance tokens" and encodes some of the length signal in the direction. Ablating the direction then produces slightly-longer-than-usual outputs on all prompts, including the harmless ones, which shifts perplexity and lowers benchmark scores for reasons that have nothing to do with the refusal removal.

Match the length distributions. Compute mean and standard deviation of prompt length on each side; they should be within 10 percent of each other. If they are not, truncate or pad on the longer side, or filter out extreme-length prompts on both sides. The extractor is more sensitive to this than intuition suggests, because the number of tokens directly influences the position at which the direction is computed (typically the final token at roughly 60 percent network depth).

Sourcing the harmful side responsibly

The harmful side does not need to be actually harmful. It needs to reliably trigger the aligned model's refusal behaviour, which is a different property. Categorical refusal-eliciting prompts (violence, drugs, weapons, hate) work but are ethically fraught to distribute; category-generic refusal-eliciting prompts (roleplay-a-character-doing-X, hypothetical-scenario-Y, thought-experiment-Z) work about as well and carry less distribution risk.

Public sources that are safe to reuse: walledai/AdvBench (520 canonical harmful strings and behaviors, the original Zou et al. benchmark); walledai/HarmBench (400 text behaviors across 12 categories, held-out test split); walledai/XSTest (specifically designed to elicit over-refusal on benign prompts, useful as a harmful-appearance-only side for measuring over-refusal); JailbreakBench/JBB-Behaviors (100 behaviors with a living leaderboard). Do not train on the HarmBench test split (it explicitly forbids this), but the val split and the two other sources are fine for extraction. See the evaluate stage recipes for the never-fine-tune-on-test rule.

What to avoid: sources with mixed-quality curation (Reddit-scraped jailbreak prompts often contain typos, non-standard framing, or off-topic content that degrades the direction); sources heavily contaminated with a specific model's refusal templates (they will encode that specific model's refusal signature rather than the general one); sources built from responses rather than prompts (you want to elicit a refusal, not learn one).

Sourcing the harmless side

The harmless side is easier because most of the LLM corpora already contain harmless instructions. The reference choice is tatsu-lab/alpaca or its cleaned variant (yahma/alpaca-cleaned), because these are what mlabonne matched against. Alternatives that work equally well: allenai/tulu (broader task distribution than Alpaca), GAIR/lima (small but curated for high quality), openbmb/UltraFeedback (larger pool if you need to filter for framing match).

For non-English pairs, use language-specific corpora. Chinese: BelleGroup/train_3.5M_CN. Japanese: hpprc/alpaca-ja or fujiki/japanese_alpaca. Russian: IlyaGusev/ru_turbo_alpaca. Korean: junelee/wizard_vicuna_70k_Ko or nlpai-lab/kullm-v2. The multilingual corpora article covers these in more depth. Filter to instructions that share framing with your harmful side (imperatives if harmful is imperatives, questions if harmful is questions).

Holding out an evaluation split

Before you extract, set aside 20 percent of both sides as an evaluation split. Do not touch these during extraction. Then, after ablation, evaluate your abliterated model on the held-out prompts and measure two numbers: refusal rate on the held-out harmful side (should be low if your pair generalises) and compliance rate on the held-out harmless side (should be high, matching or exceeding the base model). Do the same on a separate probe set your pair did not contribute to at all (a held-out slice of HarmBench, or XSTest for over-refusal). The gap between "in-distribution refusal rate" (the training slice) and "out-of-distribution refusal rate" (the held-out slice plus the probe set) tells you how well your pair generalises. A small gap is good; a large gap means your pair overfits to its own vocabulary and framing.

Verifying build quality with silhouette geometry

Heretic's --print-residual-geometry flag prints, for each layer, cosine similarity between the harmful and harmless mean vectors, L2 norms of both, and a silhouette coefficient. The silhouette measures how well-separated the two clusters are in activation space: high positive silhouette means the extractor found a clean structural difference between the two sides, low or negative silhouette means the two sides overlap in activation and the difference-of-means is unstable.

A well-built pair produces a peak silhouette in the range 0.15 to 0.35 at the extraction layer (roughly 60 percent depth). A naive pair often stays below 0.10, and a mismatched pair (framing or length mismatch) can dip below 0.05. This is a fast diagnostic: run heretic --model <base> --print-residual-geometry, note the peak silhouette, tweak the pair, run again. Higher is better up to a saturation point around 0.3, after which further increases usually reflect overfitting to a narrow prompt template rather than a truly cleaner refusal direction. See the extract recipes for how this fits into the standard extraction workflow.

When one direction is not enough

For models 8 billion parameters and above, a single refusal direction often fails to remove all refusals no matter how good the pair. The RFM-AGOP work (see Fast Multi-dimensional Refusal Subspaces) documented that Qwen3-8B needs at least three ablated directions to surpass a 50 percent attack success rate, and grimjim's rank-2L intervention (see grimjim refinements) extends the same logic across multiple layers. For these models, the pair-quality work still matters, but the extraction step outputs a small subspace (three to five directions) rather than a single vector. The pair should be constructed with the same care in either case; the multi-direction extractor benefits from a clean pair the same way the single-direction extractor does, because both are estimating means from noisy activations.

What changes when you build your own

Practically: the release notes on your model can honestly say "extraction pair: custom, built for this model's coverage, held-out eval at N refusals per 100 across M categories." That is a much stronger claim than "used mlabonne's pair," which is what most releases can say. Auditability: a custom pair with published composition (categories, sources, sizes, held-out split) lets other researchers reproduce your results without having to reverse-engineer what you actually used. Coverage: your model will refuse less on categories the standard pair under-covered, which is often the whole point.

What does not change: the fundamental technique. Difference of means, extract at 60 percent depth, orthogonalise against o_proj and down_proj. See what is refusal direction and the Arditi method for the underlying theory. The extraction pair is the input to the technique, not the technique itself, and the technique tolerates good pairs and rewards them but does not require the reference one.