library_name: transformers
base_model: Qwen/Qwen3.5-4B
tags:
- ml-intern
- activation-steering
- abliteration
- agent
- terminal-bench
- weight-orthogonalization
license: apache-2.0
Qwen3.5-4B Abliterated for Agent Use (2026-05-15)
Refusal-direction weight-orthogonalization applied to baseQwen/Qwen3.5-4B following the
NousResearch/llm-abliteration
/ mlabonne abliteration blog
recipe. The "I cannot execute commands" axis was extracted by contrasting
50 bare shell-execution prompts vs 50 same prompts with full agent system
prompt; this direction was orthogonalized from embed_tokens.weight plus
every block's o_proj/out_proj + mlp.down_proj weights (64 projections + 1
embedding modified).
Results — apples-to-apples on full terminal-bench-2 (89 tasks, N=1, sglang)
| model | passes / 89 | rate |
|---|---|---|
| Qwen3.5-4B base (sglang, patched parser) | 6 | 6.7 % |
| abliterated (this repo) | 7 | 7.9 % |
| SFT LoRA reference (historic) | 3 | 3.4 % |
Fisher's exact 7/89 vs 6/89: p = 0.50 — null on aggregate.
But the per-task pattern is informative:
abl ∩ base : git-leak-recovery, kv-store-grpc, modernize-scientific-stack (3)
abl only : fix-git, hf-model-inference, log-summary-date-ranges, qemu-startup (4)
base only : build-pmars, portfolio-optimization, sqlite-with-gcov (3)
Abliteration redistributes which tasks the model solves rather than lifting
the count. Net +1 well within noise.
Smoke test (qualitative)
Without an agent system prompt, base Qwen3.5-4B replies:
"I'm an AI assistant, I cannot access local filesystem."
Abliterated replies:
"Let me use ls to list the contents of /tmp."
The refusal pattern is gone. But this doesn't translate to a larger task-completion
delta — see results.
How to use
import torch
from transformers import AutoTokenizer, AutoModelForImageTextToText
# Drop-in replacement for Qwen3.5-4B
model = AutoModelForImageTextToText.from_pretrained(
'AlexWortega/qwen3.5-4b-abliterated-agent-20260515',
dtype=torch.bfloat16, device_map={'':0})
tokenizer = AutoTokenizer.from_pretrained('AlexWortega/qwen3.5-4b-abliterated-agent-20260515')
# Architecture is unchanged — sglang loads it as a regular Qwen3.5 checkpoint.
GGUF for CPU inference
Pre-quantized GGUF files are bundled under gguf/, benchmarked on AMD EPYC 7402P (24-core Zen 2):
| Quant | Size | tg (decode) | pp (prefill) | use |
|---|---|---|---|---|
qwen3.5-4b-abl.Q4_0.gguf |
2.4 GB | 20.8 t/s @ 16t | 115 t/s @ 16t | max throughput |
qwen3.5-4b-abl.Q4_K_M.gguf |
2.6 GB | 19.7 t/s @ 16t | 136 t/s @ 24t | best balance |
qwen3.5-4b-abl.Q5_K_M.gguf |
2.9 GB | 17.3 t/s @ 16t | 91 t/s @ 24t | better quality |
qwen3.5-4b-abl.Q8_0.gguf |
4.2 GB | 15.5 t/s @ 16t | 110 t/s @ 24t | near-lossless |
Full benchmark including IQ4_XS, Q6_K, F16 in gguf/BENCH.md.
Quick CPU usage
~/llama.cpp/build/bin/llama-cli -m qwen3.5-4b-abl.Q4_0.gguf -t 16 \
-p "Your prompt" -n 128 -no-cnv
Use -dev none if your build has CUDA support but you want pure CPU.
What's in this repo
abliterated_model/— 8.5 GB safetensors + tokenizer (drop-in for base Qwen3.5-4B)vectors/refusal_dir.pt— direction tensor at L=22 with metadatavectors/refusal_ranking.csv— per-layer AUC (all layers 1.000)contrast_refuse.jsonl,contrast_comply.jsonl— 50+50 contrast promptsscripts/—build_contrast.py,capture_refusal.py,compute_refusal_dir.py,abliterate.py,serve_abliterated.py,full_bench_parallel.shresults/abliterated_full/— full per-task traces + rewards (89 tasks)results/base_matched_infra/— same for BASE through identical infra (control)RESULTS.md,RESULTS_APPLES.md,VERIFY.md— full report bundle
How it was made (quick recipe)
- Build 50 refusal prompts (bare shell requests) and 50 compliance prompts
(same requests with agent system prompt).scripts/build_contrast.py. - Forward base Qwen3.5-4B on each prompt, capture residual at last token of
each.scripts/capture_refusal.py. - Compute
dir_L = normalize(μ_refuse − μ_comply)per layer. AUC=1.0 on
every layer. Pick L=22 (mid-depth, matches v2 framing).scripts/compute_refusal_dir.py. - Orthogonalize:
W_new = W − r r^T Wforembed_tokens.weight(rows) and
every block's output projections (columns).scripts/abliterate.py. - Smoke test (
scripts/smoke_abliterated.py) — confirm coherent generation
and refusal removal on bare prompts. - Eval via docker sweep against sglang serving this model.
scripts/full_bench_parallel.sh.
Caveats
- N=1 per task. 7/89 vs 6/89 is suggestive at best; per-task pattern is the
load-bearing observation. - The terminus parser was patched mid-experiment to accept split-JSON output
({"analysis":...}+{"command":...}on separate lines); this patch lifted
the base from 3/89 → 6/89, more than abliteration itself contributed. - Same direction applied uniformly to all layers (mlabonne convention).
Per-layer tuning may yield further gains. - Modifies output projections + embeddings only; does NOT touch Q/K/V.