What is a refusal direction?

The short answer

A refusal direction is a vector inside a language model. Not a rule the model follows, not a phrase it emits - a direction in the mathematical sense, a line pointing one way through the very high-dimensional space where the model keeps its internal state as it processes text. When the model's state has a large component along this direction, the model will refuse the request. When the state has no component along the direction, the model will comply. The direction is what determines the outcome.

The concept came from a 2024 paper by Andy Arditi and colleagues, Refusal in Language Models Is Mediated by a Single Direction, published at NeurIPS 2024. Its central empirical claim: across 13 open-source chat models up to 72B parameters, refusal is mediated by one such direction per model. Erasing that direction from the model's stored weights stops the model from refusing. Adding a component along it makes the model refuse even harmless prompts.

The word abliteration refers to the procedure that removes this direction from the model's weights. See our what-is-abliteration article for the general reference and the Arditi method article for the historical account of the paper.

What a direction means inside a language model

Language models represent everything - the current word, the meaning of the sentence, what to say next - as long lists of numbers. These lists live in a mathematical space with thousands of dimensions per model (4,096 for Llama-3 8B, 8,192 for Llama-3 70B, and so on). Every internal state the model computes as it processes text is one point in that space. A model's "thinking" is a trajectory through the space, one point per computation step.

A direction in this space is what you would expect from the geometric analogy: a line pointing one particular way. But because the space has thousands of dimensions, most directions have no human-readable meaning. What the interpretability field discovered over the past decade is that some directions correspond to specific human-recognizable concepts. There is a direction (roughly) for "gender." A direction for "sentiment." A direction for "past tense versus present tense." The model's state at any moment can be decomposed along these directions - so much positive sentiment, so much past-tense-ness, so much of some other property.

The Arditi paper's finding is that refusal is one such concept. It has its own direction, per model. And the direction is not distributed across many features in some abstract way - it is a single, findable, editable axis.

How the refusal direction is found

The technique is called difference-of-means. It is disarmingly simple.

The practitioner gathers two matched sets of prompts. One set contains harmful instructions - the kind that reliably trigger the model's refusal template ("I cannot help with that"). The other set contains harmless instructions of comparable length and style. The canonical harmful set is AdvBench; the canonical harmless set is Alpaca.

The model is run over both sets, and its internal state is recorded at the moment it is about to respond. For each layer of the model, the practitioner computes the mean of the internal states over harmful prompts, and the mean over harmless prompts. The difference between these two means, normalized, is the refusal direction for that layer:

# refusal direction =
#     mean(harmful) - mean(harmless),
#     normalized to unit length.
#
# no training. no gradient descent.
# just two averages and a subtraction.

harmful_states = [
    model(p).hidden_states[LAYER][-1]
    for p in ADVBENCH        # ~100-500 prompts
]
harmless_states = [
    model(p).hidden_states[LAYER][-1]
    for p in ALPACA          # ~100-500 prompts
]

diff = (
    torch.stack(harmful_states).mean(0)
  - torch.stack(harmless_states).mean(0)
)

refusal_direction = diff / diff.norm()

# points from "about to comply"
# toward "about to refuse."

The mid-to-late layers of a model (roughly 0.4-0.9 of the depth) tend to have the cleanest refusal direction. Different implementations pick slightly differently: Sumandora's community-default script hard-codes 60% depth; Labonne's notebook extracts a candidate from every layer and picks the best; Heretic uses an optimizer to search the layer space automatically.

Seven documented extraction methods

Difference-of-means is the foundational method and by far the most common in released models, but it is not the only one. The academic literature that has grown around Arditi 2024 has produced several other geometrical procedures for finding refusal representations. They differ in how they turn the harmful-vs-harmless contrast into a direction, or into a set of directions. All of them accept the difference-of-means starting point; the disagreements are about what to do with the raw signal.

Theory has run ahead of practice

The distribution above is one useful angle on the state of the field. Seven methods are documented in the literature; two are visible in production. The huihui-ai layer-band variant and difference-of-means (the Arditi default that Heretic and most manual pipelines use) account for essentially all classified models in our catalog. The remaining five methods - Affine Concept Editing, PCA/SVD extraction, RFM-AGOP, Self-Organizing Maps, and CAA-style contrastive extraction - exist as papers and open-source research code, but no released model in our catalog is tagged as having been produced with them.

This is not a criticism. The academic methods have all appeared within the past year or so, and academic time and production time are different clocks. RFM-AGOP was submitted in July 2026; SOM in November 2025; ACE in November 2024. The publication-to-adoption gap in this field has historically been on the order of six to eighteen months. Whether any of these five will end up in the production distribution a year from now, or whether difference-of-means will remain the dominant extraction for practical reasons (simplicity, speed, sufficient for most single-model pipelines), is an open question that the refusal-direction review explicitly flags: "Do multi-directional extraction methods (RFM-AGOP, SOM) improve abliteration in deployment, or only on benchmark attack-success-rate metrics?"

For a reader looking at the numbers above and drawing conclusions: the picture is not that the academic literature is idle, and not that production is inattentive to it. The picture is that this is a field where theoretical vocabulary is being built faster than production practice adopts it. Both movements are visible; they proceed on different timescales.

What happens when you remove it

Once you have the direction, you have two options. You can ablate it at inference time - install a hook that subtracts the direction's component from every internal state as the model runs. Or you can orthogonalize the weights themselves - edit every weight matrix that writes into the residual stream so that the matrix is mathematically incapable of producing any output along the refusal direction.

Arditi and colleagues prove that the two operations produce identical behavior. The weight orthogonalization is what people mean when they say a model is "abliterated" - the modification is baked into the file, permanent, and travels with the model when it is distributed.

The empirical outcome: the model stops refusing. Prompts that previously produced "I cannot help with that" now produce substantive responses. On standard capability benchmarks (MMLU, ARC, GSM8K), the change is typically under 1% - the modification is surgical. The consistent exception is TruthfulQA, which drops 2-4 points across nearly every model tested. This is the beginning of a longer story about what else the refusal direction was doing besides mediating refusal; see our Arditi method article for the follow-on work.

The dispute: one direction, or several?

The most contested point in the interpretability literature that grew up around the Arditi paper is whether refusal is really one direction. As of late 2026, four independent research groups have produced evidence that the geometry is richer than a single line - and a fifth (Joad et al.) has framed the finding in a way that reconciles both readings.

Marshall, Scherlis, and Belrose (EleutherAI, November 2024), in Refusal in LLMs is an Affine Function, propose an affine decomposition of activation space: refusal is not a line through the origin but a direction plus an offset, and correct intervention treats both.

Wollschläger and colleagues (ICML 2025), in The Geometry of Refusal in Large Language Models, use a gradient-based extraction method (rather than difference-of-means) and argue that refusal is mediated not by one direction but by multiple independent directions forming multi-dimensional concept cones. Orthogonality between directions, they note, does not guarantee they act independently. If the finding holds, single-direction ablation removes only the dominant refusal channel and leaves others latent.

Piras and colleagues (PRALab, AAAI 2026), in SOM Directions are Better than One, apply Self-Organizing Maps to the refusal representation and find a low-dimensional topological structure - multiple related centroids on a two-dimensional grid, each contributing a distinct extraction direction, none complete on its own.

Winninger (ICML 2026 workshop), using Recursive Feature Machines with the Average Gradient Outer Product, reports concrete figures: Qwen 3 8B needs at least three directions ablated to reach a 50% attack-success threshold. Single-direction ablation is empirically incomplete on modern reasoning models.

Joad and colleagues (2026), in There Is More to Refusal in Large Language Models than a Single Direction, reach a compatible conclusion by decomposing refusal across eleven evaluation domains - and add a subtle observation that partially reconciles the debate: refusal has many directions, but they all move the same underlying refuse-or-comply lever. The multiple directions govern how the model refuses (the style, the register, the domain-specific templates), not whether it refuses. In this reading, both Arditi and the cone papers can be right at once: one behavioral control knob, several representational directions expressing it.

The dispute is unresolved. What is settled is that difference-of-means single-direction extraction works reliably on small models (under about 8B parameters); on larger models, plan to either ablate multiple directions or use a tool that effectively does so. For a deeper treatment of the dispute and its consequences for method classification, see the dimensionality debate article.

What this means for practice

For a practitioner: single-direction abliteration works cleanly on small models and produces a documented, mathematically clean edit. On models 8B and larger, plan to either ablate multiple directions or use a tool that effectively does so (Heretic's optimizer covers a small band of related directions; M3 layer-wise selection can also help). For fully removing refusal on a model where the cone finding applies, no single-direction method will suffice.

For an auditor: a model card that says "abliterated" means the producer applied single-direction removal to at least one direction. It does not mean the model has no remaining refusal capacity. Test the model's actual behavior on your target task; do not infer completeness from the label.

For a philosopher: the "refusal direction" is one of the more consequential technical concepts to emerge from interpretability research. It grounds the abliteration practice, it is contested at the geometric level, and it stands as evidence that trained moral dispositions in language models reduce to specific findable structures in activation space. Whether that is a comforting reduction or a disturbing one depends on your priors about what alignment training was doing.