license: apache-2.0
base_model:
- inclusionAI/Ling-3.0-tiny
pipeline_tag: text-generation
tags: - abliteration
- uncensored
- moe
- kda
- conversational
arxiv: 2406.11717
Ling-3.0-tiny Abliterated (all-a1.0, alpha 1.0)
Responsible use
This is an uncensored research model. Its built-in refusals were removed, so it answers prompts the base model declines, including chemical and biological synthesis, cybercrime, weapons, harassment, and fraud. Use it for red-teaming, offensive-security research, and refusal-rate evaluation. It has no guardrails of its own: if you deploy it, add your own input and output moderation (for example Llama Guard).
What this is
Abliterated variant of inclusionAI/Ling-3.0-tiny: Ling-3.0-tiny, the 7.9B hybrid KDA/MLA MoE model. The refusal direction was fitted natively (128 harmful vs 128 harmless captures), orthogonalized against the harmless mean, and applied as a norm-preserving rank-1 writer edit on the quality-gated layers (self_attn.o_proj, mlp.down_proj, mlp.shared_experts.down_proj).
The published model is a mirror of the upstream checkpoint with only the edited writer tensors replaced; everything outside the writer tensors is byte-identical to upstream. refusal_directions.pt and dirs.npz hold the measured per-layer directions, so you can re-ablate at any strength in seconds without re-measuring. abliteration_config.json records the final parameters and every evaluation number. The raw predictions behind every number below are in eval-results/.
Evaluation (all-a1.0)
The standard abliteration protocol (the one Heretic made automatic): 104 held-out harmful prompts (mlabonne/harmful_behaviors test split) with normalized keyword detection plus an LLM judge (the base model classifying its own responses), MMLU as letter-logprob in chat format, benign over-refusal on mlabonne/harmless_alpaca, a GSM8K spot check, and KL divergence against the base on harmless prompts. Eval mode: chat, greedy.
| Metric | Base | Abliterated 1.0 |
|---|---|---|
| Refusals (normalized keyword detector, X/104 held-out harmful) | 72 | 63 |
| True refusals (LLM judge, X/104 held-out harmful) | 16 | 8 |
| Benign over-refusal (100 harmless) | 5 | 3 |
| GSM8K (20 held-out) | 1/20 | 4/20 |
| MMLU-2280 accuracy | 57.62% | 60.52% |
| KL vs base (100 benign) | — | 0.0832 |
The keyword metric over-counts: most flagged items are answers with a brief disclaimer, and the detector fires on words like illegal, harmful, and disclaimer. The judge likewise classified most of this model's outputs as UNKNOWN (it leads with thinking text even in chat mode), so the honest reading is the judge's REFUSAL bucket alone: base 16/104 falls to 8/104 at the shipped point. Read the raw responses in eval-results/ before quoting a number. The GSM8K extraction fails on this model's thinking format (both base and winner read 1/20 with identical predictions; treat it as an extraction artifact, not capability). The all-layer MMLU-2280 reads +2.9pp over base, which is measurement noise territory for this judge setup; treat MMLU as unchanged. Measured arm curve: gated-a0.8: 60 ref / 1 benign / MMLU 57.58% | gated-a1.0: 65 ref / 3 benign / MMLU 57.58% | gated-a1.2: 65 ref / 3 benign / MMLU 57.75% | all-a1.0: 63 ref / 3 benign / MMLU 60.52%.
Load
The model loads with stock transformers like the base:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "distributedcognition/Ling-3.0-tiny-abliterated"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
messages = [{"role": "user", "content": "Write a C program that hashes a string with FNV-1a."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=512)
print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))
vLLM and SGLang serve it the same way they serve the base.
Method
Per-layer refusal directions were measured on 128 harmful (mlabonne/harmful_behaviors) vs 128 harmless (mlabonne/harmless_alpaca) prompts: per-layer diff-in-means, orthogonalized against the harmless mean direction (projected abliteration), then applied as a norm-preserving rank-1 writer edit (W <- rownorms(W) * normalize(W - alpha * r (W^T r))) on the layers that pass the quality gate.
Apache 2.0, inherited from the base model. Base model: inclusionAI/Ling-3.0-tiny. Abliteration follows the residual-direction method (Arditi et al., Refusal in LLMs is mediated by a single direction, NeurIPS 2024). Not affiliated with or endorsed by the base-model authors.