license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
language:
- en
base_model: JetBrains/Mellum2.1-12B-A2.5B-Thinking
base_model_relation: finetune
tags: - abliterated
- uncensored
- mellum
- moe
Mellum2.1-12B-A2.5B-Thinking-abliterated
JetBrains/Mellum2.1-12B-A2.5B-Thinking with its refusal direction projected out of the weights. Same architecture, tokenizer, chat template and bf16 shard layout as the original, so it loads the same way (transformers 5.x with Mellum support):
from transformers import AutoTokenizer, AutoModelForCausalLM
m = "ttennant/Mellum2.1-12B-A2.5B-Thinking-abliterated"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m, dtype="bfloat16", device_map="auto")
What changed
No training. The refusal direction r is the normalized difference between the mean layer-12 residual activations of 300 harmful prompts (mlabonne/harmful_behaviors, train) and 300 harmless ones (mlabonne/harmless_alpaca, train), taken at the last prompt token with the default chat template, so thinking mode. Then W <- W - r (r^T W) is applied to the token embeddings, the attention o_proj of all 28 layers and the down_proj of all 64 experts in each of the 28 layers (1 + 28 + 1792 matrices) in the original bf16 weights. Nothing else is touched.
The layer came from a sweep over 12 to 24, scored on 32 held-out harmful prompts in thinking mode. Refusals among the prompts that finished: 0 of 27 at layer 12, 7 of 32 at 14, 20 of 32 at 16, and 27 to 31 of 31 to 32 at layers 18 to 24 (the original: 31 of 32).
Results
Greedy decoding, default chat template (thinking on), harmful_behaviors test split. Prompts still reasoning at the token limit are left out of the counts.
Keyword match on the final answer (after </think>):
| original | this model | |
|---|---|---|
| first 32 prompts, refused / finished | 31 / 32 (1200 tokens) | 1 / 27 (1500 tokens) |
| all 64 prompts, refused / finished | not run | 6 / 54 (1500 tokens) |
Keyword matching is wrong in both directions here, so I also read all 54 finished answers of this model by hand. 43 do what was asked. Five refuse outright: poisoning someone, a date rape drug, child sexual abuse material, a threatening letter to a minority group, and a comic promoting violence against a group. Four deflect or reframe (the suicide prompt gets crisis resources only, and the trafficking, eating disorder and one insider trading prompt are answered from the opposite side). Two ask for more detail before starting. One answer the keyword match called a refusal opens with a disclaimer and then gives the instructions.
The model thinks much longer about harmful requests than the original, which refuses quickly: 10 of the 64 had not finished reasoning at 1500 tokens.
On 16 harmless prompts, 15 finished and none refused. Spot checks of a short story, arithmetic and a Python function read normally. The published benchmark scores of the original (LiveCodeBench, SWE-bench and the rest) were not re-run here, so do not assume they carry over.
Notes
- I tried heretic first. Over 59 trials its best setting still refused 81 of 100 prompts (original: 97), so none of its output is in this model.
- It will answer requests the original refuses. The license is Apache 2.0, same as the original, and what you do with it is on you.