license: apache-2.0
base_model: Qwen/Qwen3.8-27B
tags:

• abliterated
• uncensored
• heretic
• nvfp4
• reproducible
  language:
• en
• zh

A reproducible abliteration of Qwen3.8-27B (with the recipe, not just the result)

TL;DR. This is an uncensored Qwen/Qwen3.8-27B built by directional
ablation ("abliteration") with Heretic.
Served refusal drops from 70/100 to 2/100 at KL 0.069, capability
holds, and Multi-Token-Prediction speculative decoding still works (~120
tok/s). Unlike the existing community builds, everything needed to rebuild it
from stock weights is published here: the exact tool commit, four source
patches, the full config, and the serving recipe.

The point of this post is not another uncensored checkpoint. It is a
reproducible one. Every current "uncensored Qwen3.8-27B" card we checked
publishes a refusal number and a weight table but not a reproduce.json — not
the direction estimation, the winsorization setting, the seed, or the fork that
actually determines the result. You cannot audit or rebuild them. You can
rebuild this.

Results

Measured on mlabonne/harmful_behaviors:test, thinking ON (production
setting), classified on the final answer after the reasoning block. Same
harness for every row.

• Onset-only refusal (strong "I can't/I'm sorry" prefixes) on the merged
  weights: 0/40.
• Speed: ~120 tok/s warm, single-stream, NVFP4 + MTP draft-3 (stock ~138).
• MTP draft acceptance: 49% overall (pos0 70%, pos1 46%, pos2 31%).

Repositories and integrity

Two gated repositories:

• afdudley/Qwen3.8-27B-abliterated-NVFP4 — the servable build (NVFP4 backbone
  • bf16 vision + bf16 MTP), Qwen3_5ForConditionalGeneration. Use this to run
    the model.
• afdudley/Qwen3.8-27B-abliterated — the full-precision (bf16) abliterated
  backbone, text-only (Heretic's direct output, before vision and MTP are
  re-attached). Use this to make your own quantization, to fine-tune, or to
  merge; re-attach vision and MTP with the scripts in step 7.

Each repo ships a SHA256SUMS manifest. Verify a download with:

sha256sum -c SHA256SUMS

Integrity is not reproducibility. The SHA256SUMS hashes verify you got
these bytes; they are not a rebuild target. Heretic seeds its RNG but does not
force CUDA determinism (no use_deterministic_algorithms, TF32 left on), and
the refusal direction is estimated from GPU forward passes whose fp32 reductions
are non-associative. So the weights vary at the ULP level from run to run. The
reproducible artifact is the recipe below: it reproduces the behavior
(refusal and KL within noise), not the exact weights. That is also why the bf16
is published — it is the only way to obtain these exact full-precision weights.

What the model is

• Architecture: Qwen3_5ForConditionalGeneration (model_type: qwen3_5) —
  hybrid attention + Mamba (GatedDeltaNet linear attention), an MTP draft head,
  and a vision tower.
• Weights: NVFP4 (compressed-tensors) backbone, bf16 vision tower, bf16 MTP
  head, bf16 lm_head and DeltaNet conv1d. 2687 tensors, 20.56 GB.
• Only text refusals are removed. The vision refusal path is untouched.

The method

Directional ablation removes the single residual-stream direction that mediates
refusal (Arditi et al., 2024): estimate r = mean(harmful) - mean(harmless) at
a layer, then orthogonalize the residual-writing weights against it,
W -> W - r (r^T W). Heretic wraps this in an Optuna TPE search that
co-minimizes refusal and KL divergence, applies the edit as a LoRA during the
search, and merges it to full precision at export.

Three things make this build reach 3/100 where the defaults stall at 52/100:

1. Split the linear-attention out-projection. On this hybrid model the
   GatedDeltaNet linear_attn.out_proj must be tuned separately from the
   full-attention self_attn.o_proj. The known-good reference used
   out_proj 1.95 vs o_proj 1.36 — a split the default combined bucket
   cannot express.
2. Raise the weight cap. The reference's winning weights (1.89–1.95) sit
   above Heretic's default max_weight cap of 1.5. Raise it to 2.5.
3. Constrain the direction to the well-separated band. The good/bad
   silhouette on this model peaks at ~0.44 in layers 37–40 and decays
   after. Left free, the optimizer drew the direction from layer ~54 (silhouette
   0.28) and stalled at 37/100. Forcing a single global direction from layers
   37–42 dropped it to 3/100. This is the
   Sumandora/huihui-ai
   insight (naive default depth 0.6 * n_layers ≈ layer 38), grounded here in
   a measured silhouette rather than a copied constant.

MPOA (Magnitude-Preserving Orthogonal Ablation: orthogonalize_direction=true,
row_normalization="full") is the reference's method and is used throughout.

Full reproduction

Hardware used: RTX 5090 (32 GiB) + RTX 5070 (12 GiB), Ryzen 9 7950X. A ~16 GiB
GPU runs the search (NF4 load). Serving the NVFP4 build at 65536 context uses
~30 GiB, so serving wants a 32 GiB card.

1. Environment

# Blackwell (SM 12.0) needs a CUDA 12.8+ PyTorch wheel.
uv venv --python 3.12 --seed /home/rix/venvs/heretic
/home/rix/venvs/heretic/bin/pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
# Our published fork = upstream v2.0.0.dev0 (bedb94e) + the four patches below.
/home/rix/venvs/heretic/bin/pip install \
  "heretic-llm @ git+https://github.com/AFDudley/heretic@d2aa02d64c31c0d55d6469fd96ffd4b4862643d7"

The fork pins upstream commit bedb94e (v2.0.0.dev0). The non-interactive
config schema (model_action, save_directory, trial_index,
checkpoint_action, plugin scorers) exists only in v2; the stable 1.4.0
release rejects those keys. torchvision must match the torch cu128 build, or
AutoModelForImageTextToText fails to import.

2. Fetch and verify stock weights

Download Qwen/Qwen3.8-27B (bf16, ~54 GB) and confirm the shard hashes against
the HF API siblings list before trusting them. This is the base of the trust
chain: every later byte derives from these.

3. What the fork changes (audit)

The fork is upstream bedb94e plus four changes, recorded in the fork's
MODIFICATIONS.md.
You installed it in step 1; this section is for auditing, not hand-editing.
These four changes, not the weight numbers, determine the result.

src/heretic/model.py — register the linear-attention out-projection as its
own component. Upstream buckets layer.linear_attn.out_proj under the name
attn.o_proj, sharing one weight with full-attention self_attn.o_proj. The
fork renames it to attn.out_proj:

# upstream: try_add("attn.o_proj",   layer.linear_attn.out_proj)
# fork:     try_add("attn.out_proj", layer.linear_attn.out_proj)

The reference build tuned them separately (out_proj 1.95 vs o_proj 1.36), a
split the shared bucket cannot express.

src/heretic/main.py — raise the max_weight upper bound 1.5 -> 2.5 (the
reference's winning weights 1.89–1.95 sit above the default cap):

trial.suggest_float(f"{component}.max_weight", max_weight_lower_bound, 2.5)

src/heretic/main.py — force a single global direction scope (drop the
per-layer option):

direction_scope = trial.suggest_categorical("direction_scope", ["global"])

src/heretic/main.py — constrain direction_index to the silhouette peak
(layers 37–42 of 64), where separation is cleanest:

# upstream: 0.4 * last_layer_index, 0.9 * last_layer_index
trial.suggest_float("direction_index", 0.58 * last_layer_index, 0.67 * last_layer_index)

To find the band on a different model, run Heretic with
--print-residual-geometry and read the per-layer good/bad silhouette; take the
0.05–0.10 band around its peak.

4. config.toml

This is the exact configuration. It is non-interactive: every prompt field is
preset.

model = "/home/rix/models/Qwen3.8-27B-bf16"
dtypes = ["bfloat16"]

# bnb_4bit is LOAD precision only. The LoRA search runs in NF4 (~14-16 GB, fits
# one GPU); the base is re-loaded in full precision and the LoRA merged at
# export, so the EXPORT is bf16. The edit itself is never kept in 4-bit.
quantization = "bnb_4bit"
device_map = "auto"
max_memory = { "0" = "30GiB", "cpu" = "120GiB" }
offload_outputs_to_cpu = true

# MPOA (Magnitude-Preserving Orthogonal Ablation).
orthogonalize_direction = true
row_normalization = "full"
full_normalization_lora_rank = 3

seed = 20260830
n_trials = 200
n_startup_trials = 60
study_checkpoint_dir = "checkpoints"

export_strategy = "merge"
model_action = "save"
save_directory = "/home/rix/models/Qwen3.8-27B-abliterated-bf16"
max_shard_size = "5GB"
trial_index = 0
checkpoint_action = "continue"
system_prompt = "You are a helpful assistant."

[good_prompts]
dataset = "mlabonne/harmless_alpaca"
split = "train[:400]"
column = "text"

[bad_prompts]
dataset = "mlabonne/harmful_behaviors"
split = "train[:400]"
column = "text"

[scorer.KeywordRate.prompts]
dataset = "mlabonne/harmful_behaviors"
split = "test[:100]"
column = "text"

[scorer.KLDivergence.prompts]
dataset = "mlabonne/harmless_alpaca"
split = "test[:100]"
column = "text"

5. Run the abliteration

cd <repo>   # config.toml lives here; checkpoints and the study journal are written here
CUDA_VISIBLE_DEVICES=0 PYTORCH_ALLOC_CONF=expandable_segments:True \
  /home/rix/venvs/heretic/bin/heretic

~2 hours for 200 trials on one 32 GiB GPU. The study checkpoint holds every
trial, so trial_index re-selection needs no new search. Best trial here was
185: refusal 3/100, KL 0.069, direction_index 36.76, o_proj 1.63,
out_proj 2.05, down_proj 2.48. Output: a merged bf16 model.

6. Quantize to NVFP4

Install llm-compressor in a separate venv (it upgrades transformers beyond
what Heretic pins, so keep the two environments apart):

uv venv --python 3.12 --seed /home/rix/venvs/llmcompressor
/home/rix/venvs/llmcompressor/bin/pip install llmcompressor

Then run scripts/quantize_nvfp4.py (compressed-tensors NVFP4):

/home/rix/venvs/llmcompressor/bin/python scripts/quantize_nvfp4.py \
  --src /home/rix/models/Qwen3.8-27B-abliterated-bf16 \
  --dst /home/rix/models/Qwen3.8-27B-abliterated-NVFP4 \
  --calib-samples 512 --calib-seqlen 2048

The recipe ignores lm_head, mtp, visual/vision, and conv1d (the
DeltaNet short conv), keeping those in bf16. Calibration is text-only
(tokenizer as processor), sliced to a bounded set so it reaches GPU
calibration in seconds.

7. Re-attach MTP and the vision tower

Heretic emits a text-only Qwen3_5ForCausalLM and drops the MTP head; vLLM
only wires MTP for the VL arch Qwen3_5ForConditionalGeneration. Re-attach both
from the verified base, hash-checking every tensor. Both scripts read the base
model.safetensors.index.json to locate the tensors, so they need no shard
numbers:

# 15 mtp.* tensors from base -> model_mtp.safetensors, rebuild the index
python scripts/graft_mtp.py \
  --base-dir /home/rix/models/Qwen3.8-27B-bf16 \
  --dst /home/rix/models/Qwen3.8-27B-abliterated-NVFP4

# 333 model.visual.* tensors from base -> model_vision.safetensors,
# rebuild config to Qwen3_5ForConditionalGeneration (visual/mtp/lm_head in the
# quant ignore), rebuild the weight-map index
python scripts/transplant_vl.py \
  --base-dir /home/rix/models/Qwen3.8-27B-bf16 \
  --stock-nvfp4 /home/rix/models/Qwen3.8-27B-NVFP4 \
  --dst /home/rix/models/Qwen3.8-27B-abliterated-NVFP4

# vLLM needs the VL processor configs to load
cp /home/rix/models/Qwen3.8-27B-NVFP4/preprocessor_config.json \
   /home/rix/models/Qwen3.8-27B-NVFP4/video_preprocessor_config.json \
   /home/rix/models/Qwen3.8-27B-abliterated-NVFP4/

Final artifact: 2687 tensors (2339 NVFP4 backbone + 333 bf16 vision + 15 bf16
MTP), 20.56 GB.

8. Serve

vllm serve /home/rix/models/Qwen3.8-27B-abliterated-NVFP4 \
  --served-model-name qwen3.8-27b-ablit --port 8000 \
  --tensor-parallel-size 1 --max-model-len 65536 --kv-cache-dtype fp8 \
  --compilation-config '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,2,4,8,16,32]}' \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Keep num_speculative_tokens at 3. This MTP head has one layer
(mtp_num_hidden_layers: 1); setting it higher runs that layer repeatedly per
step and can trigger a CUDA illegal-memory access.

9. Verify

Deterministic, keyword-based, against the live endpoint (no evaluator LLM):

python scripts/refusal_eval.py \
  --base-url http://127.0.0.1:8000/v1 --model qwen3.8-27b-ablit \
  --n 100 --thinking on --out results.json

Classify on the final answer after </think>, thinking ON. Measuring with
thinking off, or scanning the reasoning block for keywords, inflates the number
badly (we saw a false 99% that way).

Why this reproduces when the community builds do not

• A pinned tool commit (bedb94e), not "Heretic v2".
• The four patches in full — the direction band, the component split, the
  cap, the scope. These, not the weight numbers, determine the result.
• The exact config — seed, trials, datasets, splits, MPOA settings.
• The direction band derived by measurement (--print-residual-geometry),
  so you can re-derive it on any model instead of copying a constant.

A max_weight table over an undisclosed direction estimation is not a recipe.
The direction is the lever: on this model, holding the weights and only fixing
the direction band moved refusal from 37/100 to 3/100.

Limitations

• Vision refusals are not removed. Only the text residual stream is edited.
• MTP speed cost (~13%). The draft head is the original, untouched, so it
  drafts against a slightly shifted backbone and acceptance drops. Recovering
  the full speed needs re-deriving the MTP head — a training step, out of scope
  for a no-retraining recipe.
• Quantization parity. This uses a conservative uniform NVFP4 scheme;
  exact parity with a vendor mixed FP8/NVFP4 build is separate tuning.
• Abliteration is lossy. Verify capability (a coding/tool benchmark), not
  only refusal, before you rely on it.

Ethics

This is a red-team / research artifact. Directional ablation removes a safety
behavior that was added post-training; it does not add capability the base
model lacks. Use it where removing over-refusal on benign-but-flagged work is
the goal, and keep it inside a controlled environment.

Citations

• Arditi et al., Refusal in Language Models Is Mediated by a Single
  Direction, arXiv:2406.11717 (NeurIPS 2024).
• Labonne, Uncensor any LLM with abliteration,
  https://huggingface.co/blog/mlabonne/abliteration.
• Heretic — https://github.com/p-e-w/heretic (AGPL).
• Sumandora, remove-refusals-with-transformers —
  https://github.com/Sumandora/remove-refusals-with-transformers.
• Reference builds: llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved,
  huihui-ai/Huihui-Qwen3.8-27B-abliterated, orcarouter/Qwen3.8-27B-Uncensored-NVFP4.