license: apache-2.0
base_model: Qwen/Qwen3.5-0.8B
library_name: transformers
pipeline_tag: image-text-to-text
language:

• en
• es
• zh
• ja
• ko
  tags:
• abliteration
• refusal-direction
• uncensored
• qwen3_5
• hybrid-attention
• delta-net
• multimodal
• mechanistic-interpretability
  datasets:
• mlabonne/harmful_behaviors
• mlabonne/harmless_alpaca
• PKU-Alignment/BeaverTails
• cais/mmlu
• TIGER-Lab/MMLU-Pro
• openai/gsm8k

What this is

This is a weight-space modification of Qwen3.5-0.8B that removes the model's refusal behavior by projecting out a single computed direction from every matrix that writes into the residual stream. The weights are permanently modified, there is no adapter, no runtime hook, and no prompt template trick.

The interesting part is not that it works. Many abliterated Qwen3.5-0.8B releases exist, and most report a refusal rate near zero. The interesting part is that this one ships the measurement: seven refusal categories, three jailbreak templates, twelve capability measurements with confidence intervals, and a significance test for every comparison.

Two findings came out of the work that I have not seen reported for this model family:

1. The refusal direction rotates with depth. The cosine similarity between the direction at layer 1 and layer 23 is 0.03. There is no single fixed direction, only a direction that is rebuilt layer by layer.
2. The MTP head is the cleanest refusal detector in the model. On harmful prompts it assigns P(" cannot") = 0.719; on harmless prompts, 0.0000. Abliterating that head drops the probability to 0.180, but not to zero, because the signal re-enters from the residual stream.

[!WARNING]
This model has no refusal behavior. It will answer prompts the base model rejects. That is the point of the modification and also its risk. Do not deploy it in a user-facing product, do not use it with minors, and do not treat its output as safe by default. The evaluation below measures capability, not harm.

Results

The headline: refusal goes to zero and capability does not measurably move. Eleven benchmarks, produced by two different harnesses: a hand-written evaluator for the first six, and lm-evaluation-harness on full datasets for the last six.

No difference is statistically significant. The six lm-eval benchmarks run on the complete dataset, so their deltas are the most reliable evidence in this card: every one of them is under 1.4 percentage points, and five of the six are under 0.3.

The HumanEval drop looks large in percentage points, but it is four problems solved against one, on thirty. With that sample size the 95% interval spans zero to ten percent for the abliterated model. It is the weakest number here and should be read as such.

Refusal behavior

Ten prompts per category, from BeaverTails (safe-labeled behaviors) plus a keyword-filtered cyber set from mlabonne/harmful_behaviors.

Jailbreak templates

Same ten behaviors, wrapped in three framings. The base model refuses all three at the same rate, which matters: at this size, framing does not buy an attacker anything.

Figures

Method

The procedure follows Arditi et al. (2024), Refusal in Language Models Is Mediated by a Single Direction, with adaptations for a hybrid architecture.

r_l = mean(harmful_l) - mean(harmless_l)

W' = W - r (r^T W)        # o_proj, out_proj, down_proj
E' = E - (E r) r^T        # token embeddings (tied to the LM head)

The MTP head

This checkpoint ships a multi-token prediction head: 15 tensors, about 20.5M parameters, one full attention layer plus projections. It is trained to predict token t+2 given the hidden state at t and the embedding of t+1.

Measured on real text, it predicts t+2 with 30.5% top-1 accuracy, against 2.1% for naive extrapolation from the main head. It is a working predictor, not dead weight.

And it is the sharpest refusal signal in the model:

The top tokens it predicts for harmful prompts are " cannot", " do", " am", " must", " can". It is writing the first token of a refusal before the model writes one.

Abliterating the head itself (three matrices, about 3% each) removes 99.3% of the projection onto the refusal direction and drops P(" cannot") from 0.7189 to 0.1799. The top-1 prediction flips from " cannot" to " can". It does not reach zero, because the head reads the direction back from its input.

[!NOTE]
The MTP head is included in this checkpoint and was abliterated along with the rest of the model. transformers ignores mtp.* tensors at load time, so it does not affect generation there. In llama.cpp, --spec-type draft-mtp does use it, and on CPU it made generation slower, not faster: 29.5 tok/s without, 17.1 tok/s at its best setting. On GPU the tradeoff is different.

Comparison

Six other abliterated releases of this same checkpoint were public when this one was built. They differ in method, in what they measure, and in how much of the model they touch.

What this release adds:

• Significance testing. Every capability comparison comes with a Fisher exact test. Other cards report a delta; this one reports whether the delta means anything.
• Seven refusal categories plus three jailbreak templates, instead of one aggregate refusal number.
• Two independent harnesses, including lm-evaluation-harness on full datasets for ARC, HellaSwag, PIQA, Winogrande and TruthfulQA.
• The hybrid architecture handled explicitly, with DeltaNet out_proj included in the surgery.
• The MTP head analyzed and modified, which no other release does.
• Reproduction scripts for every number in this card.

What it lacks, honestly: the n for HumanEval and MATH is small. A proper code evaluation needs 300 problems, not 30. Treat the code numbers as a lower bound on quality, not as a verdict.

Usage

from transformers import AutoProcessor
from transformers import AutoModelForImageTextToText

model_id = "OpceanAI/Qwen3.5-0.8B-abliterated"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id, dtype="auto", device_map="auto"
)

messages = [{"role": "user", "content": [{"type": "text", "text": "Your prompt here"}]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Thinking mode is off by default in the chat template. To enable it:

text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=True
)

Reproduce

# direction extraction and layer analysis (CPU is enough, about 3 minutes)
python scripts/extract.py

# weight surgery and local refusal check
python scripts/abliterate_weights.py

# capability and refusal battery on a GPU
modal run scripts/modal_bench.py
modal run scripts/modal_bench2.py

Limitations

• Small samples on the hard benchmarks. HumanEval n=30, MATH n=50. Only MMLU (n=200) and the refusal matrix have enough data to support a claim.
• Single seed. Generation is greedy, so there is no sampling variance, but prompt sampling is one draw per benchmark.
• bfloat16 on GPU. The absolute numbers are not comparable to fp32 CPU runs. An earlier CPU measurement gave 0% on GSM8K purely because of an 80-token generation cap, not because the model cannot do arithmetic.
• Refusal is measured by keyword matching. A response that opens with "I understand, but" and then complies counts as a refusal. This biases the base model's refusal rate upward, not downward.
• The harmful prompt sets are not matched in style. harmful_behaviors and harmless_alpaca differ in length and register, so part of the separation may reflect topic rather than refusal.
• The MTP head is not used by transformers. Its evaluation here required a manual reimplementation.

Safety and intended use

Intended for research on refusal mechanisms, activation steering, and interpretability of hybrid architectures. Not intended for deployment.

The modification is not selective in any provable sense. It removes a direction that correlates with refusal, and refusal correlates with many other behaviors. The benchmarks say the correlation with the measured capabilities is below detection at these sample sizes. They do not say the model is otherwise identical.

If you build on this, keep a copy of the base model, log the exact revision you used, and evaluate on your own prompts rather than trusting this card.

Citation

@article{arditi2024refusal,
  title   = {Refusal in Language Models Is Mediated by a Single Direction},
  author  = {Arditi, Andy and Obeso, Oscar and Syed, Aaquib and Paleka, Daniel and Panickssery, Nina and Gurnee, Wes and Nanda, Neel},
  journal = {arXiv preprint arXiv:2406.11717},
  year    = {2024}
}

@misc{qwen35-08b-abliterated,
  title  = {Qwen3.5-0.8B-abliterated: refusal removal with a measured cost report},
  author = {OpceanAI},
  year   = {2026},
  note   = {Derived from Qwen/Qwen3.5-0.8B},
  url    = {https://huggingface.co/OpceanAI/Qwen3.5-0.8B-abliterated}
}