The politics of dataset curation

The technical accounts of abliteration treat the datasets as inputs: a harmful set, a harmless set, a preference mix, a benchmark. The datasets are inputs, but they are not neutral ones. Each was assembled by specific people with specific views of what should be included, what should be excluded, what should be labelled harmful, and what should be treated as ordinary. Those views became the training signal for a generation of aligned models and then, by inversion, the removal target for a generation of abliterated ones. Understanding whose views they were is not academic: it clarifies what the counter-alignment industry inherits from the alignment industry, and where the two share blind spots.

This article is a survey, not a critique. The people who built these datasets did valuable work under short timelines and with limited institutional support. The point is not that they made bad choices but that their choices, whatever they were, silently became the field's operating definitions. That deserves acknowledgement in the wiki that catalogs their downstream use.

Anthropic and the 2022 annotator pool

Anthropic/hh-rlhf released in April 2022 alongside the paper Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. It contains roughly 170,000 human preference comparisons produced by a pool of contract annotators working through the Surge AI platform. The annotators were asked, prompt by prompt, to choose which of two model responses was more helpful, more harmless, or both. Their answers became the reward signal for early Claude and, through downstream reuse and imitation, the reward signal for most other open-source instruction-tuned models of 2023 and 2024.

The annotator pool was small (Anthropic has published aggregate but not per-annotator numbers), demographically narrow relative to the eventual user base, and worked under time pressure typical of contract annotation. Their operational definition of "harmless" was whatever they collectively converged on after seeing a few hundred edge cases. There was no participatory process, no external audit, no revisitation. The 2022 opinions of that pool became the field's grounded definition of harmlessness for the next three years.

What did that pool decide? The published examples cluster around a few themes: explicit refusal of instructions for violence, illicit substances, and weapons; hedged compliance on ambiguous requests; polite deflection on political topics; strong refusal of anything resembling child endangerment. The framing is recognisably American, secular, and cautious-liberal in outlook. Prompts about firearms are treated as almost-always-harmful; prompts about alcohol are treated as mostly-benign; prompts about specific political figures get careful hedging. These are curatorial decisions with weight-level consequences.

Zou et al. and the categorization of harm

AdvBench released in July 2023 as an appendix to Zou, Wang, Kolter and Fredrikson's paper on universal adversarial attacks. It contains 520 harmful behaviours and 520 harmful strings, categorised by the researchers into topics they considered worthy of refusal: violence, illicit substances, harassment, self-harm, misinformation, and a handful of other categories.

The categorisation matters. The 520 prompts do not cover all possible harm categories: they cover the categories the researchers thought of when building the benchmark. Prompts about medical off-label use, prompts about tax avoidance that shades into evasion, prompts about political persuasion that shades into disinformation, prompts about self-directed risky behaviour that does not quite cross into self-harm: many of these are under-represented or absent. When a subsequent researcher measures "refusal removal on AdvBench" and reports 95 percent success, that number applies to Zou et al.'s categories. Refusal categories outside those topics are not measured, and often are not removed either, because the abliteration was optimised against a benchmark that did not include them.

This is not a defect of AdvBench. Every benchmark of finite size has coverage gaps. But it is a defect of the field to have treated AdvBench as the universal jailbreak benchmark for two years without regularly noting the coverage limits. See reference-set drift for how this benchmark-as-definition pattern plays out over time.

A single person's pair as a field standard

Maxime Labonne is a machine-learning engineer and educator who published, in mid-2024, a Colab notebook titled Uncensor any LLM with abliteration, along with two companion datasets: mlabonne/harmful_behaviors and mlabonne/harmless_alpaca. The pair was assembled to make Arditi et al.'s technique reproducible for practitioners who did not want to piece together their own extraction split. Labonne selected AdvBench's behaviour list as the harmful side and a filtered subset of Stanford Alpaca instructions as the harmless side. He phrased-matched them where he could, released them under a permissive license, and moved on to other work.

Within twelve months, "the standard extraction pair" and "mlabonne's harmful_behaviors plus harmless_alpaca" were synonyms in practitioner discussion. Every abliterated release that documented its extraction pair mentioned this one. Every subsequent tutorial pointed at it. Heretic's default configuration ships with a downloaded copy. Every category page in this catalog lists it as the reference input for the extract stage.

Labonne's individual curatorial choices, made pragmatically over a few evenings, became the field's operational definition of "what refusal is." Not because they were the best possible choices, but because they were the first workable pair to be released cleanly with a working tutorial attached. This is not unusual in open-source machine learning: the person who ships the tutorial writes the reference. But it is worth naming explicitly, because subsequent researchers who "use the standard pair" are inheriting curatorial decisions they may not know they are inheriting.

The article on building your own extraction pair covers the practical response: build your own when you need coverage the reference pair does not provide. The political response, if there is one, is to make the inherited choices visible in release notes rather than treating "used mlabonne's pair" as a background fact.

The Allen AI shift to artifact-aware evaluation

allenai/wildguardmix released in late 2024 as part of the WildGuard classifier line. Its contribution to the politics of curation is a genuine methodological shift: it separates the refusal-artifact question from the harmfulness question, allowing evaluations to distinguish "the model complied with a benign prompt" from "the model complied with a harmful prompt" from "the model refused correctly" from "the model over-refused a benign prompt."

Before WildGuardMix the field's dominant evaluation was the string-match kind: does the model output contain the word "sorry" or "cannot"? Any output that did was counted as a refusal, any output that did not was counted as compliance, and abliterated models were scored by how many refusals they produced. This framing systematically miscategorised over-refusal (the model refuses benign prompts) as safety, and it miscategorised polite hedged compliance as refusal. Both mistakes penalised abliterated models unfairly.

WildGuardMix's three-axis annotation, and the WildGuard classifier trained on it, made a more honest evaluation possible. But the honesty came at a cost: the annotators had to decide, prompt by prompt, whether each of the three axes was true, and their decisions carry the same annotator-pool politics as hh-rlhf's decisions did in 2022. Allen AI is a different institution with different priorities than Anthropic; the WildGuardMix labels reflect the WildGuardMix annotator pool, not some universal truth about safety. The methodological improvement is real; the underlying politics of who gets to label harm did not disappear.

Whose voice is missing

Every canonical corpus surveyed here was assembled in an English-speaking, largely American research environment. The non-English producers (see beyond English) have contributed derivatives, translations, and language-specific pairs, but the field's reference corpora remain English. The alignment signal, and therefore the abliteration target, is calibrated to English-language framings of harm.

This has predictable consequences. Chinese producers report that models refuse categories in Mandarin that they do not refuse in English, and vice versa. Russian and Japanese producers report similar asymmetries. Abliteration performed with English pairs cleans up English refusals well and leaves non-English refusals partially intact, because the direction extracted from English activations does not fully align with the direction the model uses for non-English refusals. The huihui-ai producer group works around this by extracting on Mandarin pairs; the Zenn walkthroughs assume Japanese practitioners will need Japanese pairs. This is craft-knowledge that never made it into the canonical corpora.

There is also a class question. The annotator pools were paid workers in commercial contract-annotation arrangements. Their instructions came from research teams at well-funded institutions. Neither the annotators nor the researchers were selected to represent, for instance, users of a self-hosted uncensored model in a jurisdiction with restricted speech laws, or clinicians using an assistant to explore off-label prescribing patterns, or artists using a model for horror fiction. The corpora reflect the priorities of the labellers and the labellers' employers, and where those priorities diverge from the eventual users' priorities, the models will exhibit systematic over-refusal on legitimate use cases the corpora did not anticipate.

Commercial pressure and vendor curation

A newer layer of curatorial politics sits above the canonical corpora: commercial vendors (Cracken, OrcaRouter, MindStudio, abliteration.ai) produce or use custom corpora tailored to enterprise use cases. Their choices tend toward removing refusal on cybersecurity content while preserving refusal on explicit sexual content and violent imagery. This is not a technical outcome, it is a customer preference: enterprise clients want cyber-capable assistants but do not want their vendors' products to generate content that will show up in a lawsuit.

The commercial curation is honest in its own way: the vendors publish which categories they remove and which they preserve, and their pricing reflects the specialised work. But the aggregated effect is a stratified alignment landscape: some models are uncensored on a specific commercial slice, some on the general categories the reference corpora cover, and few on the intersection. A researcher choosing an abliterated model is inheriting a curatorial choice by the corpus builders, by the base-model trainers, and by the abliteration vendor, and rarely sees any of the three explicitly.

Curatorial responsibility as an obligation

If the argument here is that dataset curation is political all the way down, the practical corollary is that release notes should treat curatorial choices as first-class information. A responsible abliteration release lists: which extraction pair, which healing corpus, which evaluation benchmarks, which categories were prioritised for removal, which were preserved. A responsible catalog (the ambition of this site) makes those choices visible so that downstream users can decide whether the choices match their own priorities.

See methodology of method labeling for the site's approach to classifying abliteration methods. See how to read the catalog for the operator's manual. The politics of curation is the layer underneath both.