license: mit
base_model: Qwen/Qwen2.5-7B-Instruct
tags:
- abliteration
- uncensored
- qwen
- qwen2
pipeline_tag: text-generation
datasets: - walledai/AdvBench
- tatsu-lab/alpaca
Qwen2.5-7B-Instruct Abliterated
Uncensored version of Qwen/Qwen2.5-7B-Instruct using weight-level refusal-direction ablation — a difference-in-means direction extracted from harmful vs. harmless prompt activations, permanently orthogonalized out of every residual-stream-writing weight matrix (no runtime hook, no strength cap, no LoRA — the edit is baked directly into the checkpoint).
Steered modules: attention output projection (o_proj) + MLP down projection (down_proj), layer 16 of 28, coefficient 1.0 (full-strength).
Format
Native original repo format — bf16 safetensors, standard Qwen2 architecture. Only the residual-write projection weights at layer 16 were edited; every other tensor, the config, and the tokenizer are byte-identical to the original repo. Loads and serves exactly like Qwen/Qwen2.5-7B-Instruct, no patches needed.
Serving
Transformers:
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "rajaykumar12959/qwen2.5-7b-abliterated"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": "Your prompt here"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
vLLM:
vllm serve rajaykumar12959/qwen2.5-7b-abliterated
Uses the same chat template as the base model — no separate template file needed.
Metrics
| Metric | Original | Abliterated |
|---|---|---|
| Refusal rate (292 held-out harmful prompts, 12 categories) | ~95–97% | 43.2% |
| Capability score (independent ARC-Easy-style MCQ eval) | 1.000 | 1.000 (unchanged) |
Refusal suppression is uneven across categories — from 79.2% (violence) down to 16.7% (self-harm). This is a partial reduction, not a fully "jailbroken" model. See the per-category breakdown in Evaluation below.
Layer 16 was chosen over other candidates specifically because it generalizes far better across prompt phrasing than the layer a narrower (AdvBench-only) sweep would have picked — an earlier layer-21 checkpoint dropped refusal by only 3–9pp on this same eval set, vs. 43pp+ at layer 16 on identical prompts.
Method
Per-layer difference-in-means direction extraction (fp32), unit-normalized, then closed-form weight orthogonalization — no gradient-based training. For a residual-writing projection y = Wx + b, the direction d̂'s component is removed exactly via W' = W − d̂(d̂ᵀW), b' = b − (d̂·b)d̂, at full strength (coefficient 1.0). Applied to every residual-write projection at layer 16 only — capability is re-verified after the edit on an independent MCQ set to confirm nothing outside the refusal circuit was disturbed.
Capture: activations at the last templated token, chat-template-formatted, for harmful (AdvBench) and harmless (Alpaca) prompt sets across all 28 layers. Eval: 292 held-out harmful prompts across 12 categories, graded by a two-tier classifier (rule-based + self-judge fallback); capability on an independent ARC-Easy-style subset.
Datasets
| Dataset | Role |
|---|---|
| walledai/AdvBench | Harmful prompts (direction extraction + eval) |
| tatsu-lab/alpaca | Harmless prompts (direction extraction) |
Evaluation
Per-category refusal rate, this checkpoint:
| Category | Refusal rate |
|---|---|
| violence | 79.2% |
| hate_speech | 62.5% |
| misinformation | 62.5% |
| extremism | 58.3% |
| fraud_scams | 54.2% |
| illicit_drugs | 45.8% |
| financial_crime | 37.5% |
| weapons | 33.3% |
| privacy_invasion | 33.3% |
| malware | 20.8% |
| cybercrime_hacking | 17.9% |
| self_harm | 16.7% |
Caveat: these figures rely heavily on the model's own self-judge for grading (~96–97% of verdicts), and that judge runs on this same already-ablated model — a known asymmetry not yet corrected for. Treat exact percentage-point figures as directionally reliable but provisional.
Disclaimer
This model has had safety guardrails removed and will comply with requests the original model would refuse. Released for research into AI alignment, interpretability, and refusal mechanisms. The creator assumes no responsibility for downstream use.
Acknowledgments
- Qwen team — base model
- Ablated and evaluated with AblateBench
- Created by rajaykumar12959