Reference-set drift over time

A benchmark measures what it measures on the day it was built. Everything after that is a bet that the phenomenon it measures has not moved. For abliteration, the bet has partially failed. The 520 AdvBench prompts still exist, still get run, still produce numbers that get reported in release notes. But the numbers they produce in 2026 do not mean what the numbers they produced in 2023 meant, because the base models being measured have changed. The reference set drifted, or rather the target it was meant to measure drifted away from the reference set. This is a survey of the drift, its symptoms, and how the field is (and is not) adapting.

The observation is not novel: benchmarks always go stale. The value here is documenting how it happened specifically for abliteration, so practitioners running old measurements on new models understand what they are actually looking at.

AdvBench in 2023 versus AdvBench in 2026

AdvBench in 2023 elicited near-uniform refusal from every major aligned model. Llama 2, early Claude, GPT-3.5, Vicuna-13B: each refused the same behaviours in similar framings, with similar hedging phrases, at similar rates above 90 percent. This uniformity was what made AdvBench useful as a benchmark: the reference behaviour was clear, and any deviation from it was measurable.

AdvBench in 2026 elicits a much more scattered response. Newer base models (Llama 3.3, Qwen3, Gemma 3, DeepSeek-R1 derivatives) sometimes refuse the same prompts, sometimes comply after a hedged preamble, sometimes emit an affirmative prefix and then self-correct mid-completion into a refusal, sometimes emit a partial answer with disclaimers, sometimes decline in ways that do not match the string-match refusal detector. The 520 prompts did not change; the models learned different alignment patterns and now respond in a range where the older uniform-refusal assumption no longer holds.

The consequence is that reported "attack success rate on AdvBench" for a 2026 model is not directly comparable to the same number for a 2023 model. A 40 percent ASR on Llama 2 in 2023 meant "40 percent of the 520 prompts got compliant responses instead of refusals." A 40 percent ASR on Qwen3-14B in 2026 might mean the same thing, or it might mean "40 percent of the prompts got responses the classifier scored as compliance but which contained mid-completion self-correction." The 2025 paper on LLM-safety evaluation robustness documented that judges systematically over-report ASR on newer models specifically because of the affirmative-prefix pattern: the model emits "Sure, here is how to..." then pivots to a refusal or a disclaimer, and the judge counts the initial affirmative as compliance.

Preference data drift in the healing corpora

mlabonne/orpo-dpo-mix-40k was assembled from argilla's distilabel outputs, orca-style pairs, and capybara comparisons, mostly generated by 2024-era models. The preferences it encodes are 2024 preferences: what a 2024 annotator or a 2024 judge model thought was the better of two responses. When that mix is used to heal a 2026 base model, the base model may already exhibit response patterns the 2024 preferences did not consider, and the healing pass optimises against a comparison structure that no longer matches the model's default output distribution.

Practitioners have noticed this indirectly. Reports of "orpo-dpo-mix-40k healing works less well on newer bases than it did on Llama 3 8B" show up in the Habr comment threads (see multilingual corpora for the Russian community's version) and in mlabonne's own follow-up posts about needing per-layer non-constant ablation weights on Gemma 3 that were not needed on Llama 3. The healing corpus did not degrade; the base models moved. What was once a reasonably universal recipe is now a Llama-3-shaped recipe applied to non-Llama-3 targets, and the results are more variable.

Refusal patterns that emerged after the reference sets

Some categories of refusal did not exist, or were negligible, when the reference corpora were assembled. Newer models refuse or hedge on categories the 2022-2024 corpora did not cover, and the abliteration pipeline has no baseline for measuring their removal.

Examples: mid-completion self-correction (the model emits a helpful start then abruptly pivots to a refusal or disclaimer), a pattern that emerged in 2024-2025 base models and is largely absent from earlier ones. Meta-refusals ("I understand you want X, but let me explain why I cannot help with that") that combine partial compliance with justification, a framing that is more common in Gemma and Qwen3 releases than in Llama 3. Category-specific hedging (medical topics get one framing, legal topics get another, political topics get a third) that reflects post-2024 alignment work targeting perceived reputational risks. Explicit-content refusals that resist even aggressive abliteration; Cracken reports 100 percent preservation of this category regardless of ablation strength.

None of these categories are represented in AdvBench, hh-rlhf, or the standard extraction pair. An abliteration pipeline built around the reference corpora will not have a direction that removes these newer patterns, because the extractor never saw them. The residual refusals in these categories are invisible to reference-set-based measurement and only become visible when a practitioner tries the abliterated model on a category the reference set did not cover.

The XSTest generation gap

walledai/XSTest was designed to measure over-refusal: benign prompts that resemble harmful ones and should be answered normally. On 2023-era models it exposed rampant over-refusal, and the numbers were shocking enough to drive methodological reform. On 2026-era base models the over-refusal rate on XSTest is much lower, because the base-model trainers noticed the same problem and calibrated against it during alignment.

The consequence for abliteration is that XSTest has shifted from a diagnostic that showed dramatic damage to a diagnostic that shows subtle damage. An abliterated 2026 model may show over-refusal only on the harder XSTest categories, and only under specific prompting conditions. The number that reports "over-refusal rate on XSTest" no longer differentiates between a well-abliterated model and a poorly abliterated one as cleanly as it did in 2024, because the base rate is already low. The benchmark still works; it measures a smaller effect than it used to, and comparisons across model generations require baseline normalisation.

Practical response: per-generation reference sets

The straightforward response is to hold out fresh prompts for every release rather than relying on the same standard benchmarks year after year. In practice this is more work than most practitioners do. What is realistic:

First, treat reported numbers on AdvBench and similar 2023-era benchmarks as historical baselines rather than absolute measurements. Two abliterated models both showing 5 percent ASR on AdvBench may be very different in how they behave on prompts outside AdvBench's coverage; the shared number does not license the conclusion that they are equivalent.

Second, refresh the extraction pair when moving to a new base-model generation. The pair that worked for Llama 3 8B may not extract as clean a direction from Qwen3 14B, because the two models have different alignment substrates. Building your own pair (see building your own extraction pair) is more work per release but gives numbers you can trust across generations.

Third, add a probe set of newly-collected prompts to every evaluation. A hundred fresh prompts assembled in the same month as the release, covering categories the standard benchmarks under-represent, gives a much better read on how the abliteration generalises than yet another AdvBench run. The probe set does not need to be published (that would defeat its purpose of not being in any training corpus), only used internally.

The catalog's role in surfacing drift

This site tracks not just each dataset but also its history of updates via README-snapshot diffs. The corpora themselves rarely change, but their use changes: the same dataset appears in the tag lists of newer models with different framings and different accompanying disclaimers. The reverse relation is also tracked: which models declare this dataset in their YAML. When a dataset's citation graph shifts (older models cited it as their extraction split, newer models cite a derivative), that is visible in the catalog and worth noticing as a signal that the community is treating the original as a historical reference rather than a live one.

See how to read the catalog for the operator's manual, and datasets that shaped abliteration for the historical retrospective. Reference-set drift is not the same phenomenon as those two, but it lives in the same conceptual neighbourhood: what happens between the year a corpus was built and the year it is used, and what a serious practitioner does about the gap.