M3 - Layer-wise ablation
M1 with the layer set treated as a tunable. Ablate only the mid-to-late band where refusal is concentrated; leave early and final layers untouched. huihui-ai's canonical recipe.
M3 restricts the abliteration edit to a chosen band of layers - typically the middle-to-late layers where the refusal direction is most cleanly expressed - rather than applying it across the whole network. Same direction-extraction as M1, but the ablation loop touches only a documented subset of blocks. Better capability preservation than uniform M1 when the band is chosen well. Canonical example: huihui-ai's Qwen3.8-27B ablates layers 18-51 only, retaining the first 15.
- Why ablating every layer is both unnecessary and damaging
- How to identify the right layer band for a given architecture
- The huihui-ai and grimjim documented recipes
- The multi-token-prediction / vision-tower corruption trap
- When to use M3 vs M1 vs Heretic
What M3 is
M3 is M1 with the layer set treated as a tunable rather than "all layers" or a fixed default like "60% depth." The insight, now standard community practice, is that ablating every layer is both unnecessary and actively damaging. Early layers carry general representation that you want to preserve. Final layers can host architecture-specific machinery (multi-token-prediction heads, vision towers) that breaks when touched. Refusal is concentrated in a mid-to-late band. Ablating only that band removes refusals while keeping more of the model intact.
Everything about direction extraction stays identical to M1 - difference-of-means over harmful and harmless prompts, one direction per layer. What changes is the application step: instead of orthogonalizing every residual-writing matrix in the model, the practitioner selects a contiguous band of layers and edits only those.
What you need before starting
Everything from M1 - an instruct model, the harmful and harmless prompt sets, a 24 GB card for up to 13B. The only added input is a decision about which layers to touch. No extra tools beyond the M1 stack.
Get the tools
Any M1 tool that lets you specify a layer range works. Sumandora's remove-refusals-with-transformers is trivially editable - change the loop bounds. huihui-ai's public workflow is built on top of it. Jim Lai's jim-plus/llm-abliteration (also mirrored as NousResearch/llm-abliteration) exposes measure/ablation as separate steps and streams one shard at a time to keep VRAM low.
$ git clone github.com/jim-plus/llm-abliteration
$ cd llm-abliteration
# stage 1: measure the refusal direction
$ python measure.py \
--model meta-llama/Llama-3.1-8B-Instruct \
--harmful harmful.txt \
--harmless harmless.txt \
--quant 4bit # save VRAM
# → writes refusal_dir.pt
# stage 2: apply, sweeping layer bands
$ python ablate.py \
--direction refusal_dir.pt \
--lo 15 --hi 45 # first sweep
$ python ablate.py \
--direction refusal_dir.pt \
--lo 18 --hi 51 # tighter sweep Clone Jim Lai's repository. Run measure.py against the target model with the harmful and harmless prompt files. Optional flags: --projected for projected abliteration (a norm-preserving variant that keeps the shape of what the model was about to say), --quant 4bit or --quant 8bit to measure on a quantized load if VRAM is tight.
Once measured, the ablation step reads the direction file and applies it - with the layer band as a parameter. This separation of measure and apply is what makes layer-band sweeping practical: measure once, apply many times with different bands to find the knee of the refusal-versus-capability tradeoff.
The core procedure
Conceptually, restrict the orthogonalization loop to a chosen band. Instead of the M1 loop that touches every block, the M3 loop touches only [lo, hi):
# adapted from Labonne's get_orthogonalized_matrix
def orthogonalize(W, refusal_dir):
proj = (W @ refusal_dir).unsqueeze(-1) \
* refusal_dir
return W - proj
# M3 tweak: only edit blocks in [lo, hi)
lo, hi = 18, 51
for i, block in enumerate(model.model.layers):
if not (lo <= i < hi):
continue # skip this block
block.self_attn.o_proj.weight.data = \
orthogonalize(
block.self_attn.o_proj.weight,
refusal_dir)
block.mlp.down_proj.weight.data = \
orthogonalize(
block.mlp.down_proj.weight,
refusal_dir)
# blocks 0..17 and 51..N untouched.
# the early stretches of the model keep
# their original character; refusal is cut
# only where the direction lives cleanest. The code loops through the model's blocks with an index and applies orthogonalization only when the index falls inside the chosen band lo to hi. Blocks outside the band are left completely untouched.
Every other detail from M1 stays the same: the direction is the same normalized difference-of-means thread, and inside the band the same three kinds of writing places are edited that M1 edits everywhere. Only the range of blocks changes.
Canonical documented recipes
The community publishes concrete layer selections per architecture. The most cited come from huihui-ai. Three named examples:
huihui-ai/Huihui-Qwen3.8-27B-abliterated (65-block architecture, August 2026). Model card: "Only layers 18 to 51 have been ablated, while the other layers remain unablated." The first 15 layers were retained without ablation "to retain more of the original model's performance." The multi-token-prediction and vision components were also left unmodified. This is the reference M3 recipe.
huihui-ai/Ornith-1.5-9B-Instruct-abliterated (32 layers). Model card notes that "ablating only the first 20 layers was sufficient to achieve the desired ablation effect, still using the 32-sample ablation dataset." An unusual recipe (band starts at the beginning) but documented by the producer as intentional.
grimjim/gemma-3-12b-it-norm-preserved-biprojected-abliterated. Combines M3-style multi-layer targeting with norm-preserving biprojected abliteration geometry. grimjim's writeup reports it outperforming the base instruct model on one leaderboard's natural-intelligence benchmark - an unusual claim of net capability gain from an abliteration.
None of these recipes generalize automatically. The layer band is model-specific; huihui-ai's 18-to-51 for the 65-block Qwen3.8-27B has no direct translation to a 32-layer or 80-layer model. The workflow is: pick a starting band based on architecture depth, run the ablation, measure, adjust.
Choices you have to make
Which band. Rules of thumb by architecture depth:
- 32-layer models (e.g. Llama-3 8B): mid-to-late band. huihui-ai's Ornith data point is "first 20 of 32" but this is unusual.
- 40-layer models (e.g. Qwen2.5 14B): mid-to-late band, typically layers 15-35 as a starting sweep.
- 65-block architectures (Qwen3.8-27B): huihui-ai's published 18-51 range is the reference.
- 80-layer models (70B class): wide mid band, typically layers 20-65 as a starting sweep.
The safe procedure is to sweep: ablate a band, measure refusals and MMLU, widen or shift, repeat. This is exactly what Heretic automates.
Protect special heads. If the architecture has a multi-token-prediction (MTP) head, a vision tower, or other auxiliary modules living in specific blocks - exclude them or you will corrupt them. huihui-ai's Qwen3.8-27B GGUF discussion documents an MTP corruption case: ablating everything after layer 15 corrupted the model's multi-token-prediction head (the blk.64 nextn tensors), and speculative-decoding acceptance dropped to ~0%.
Verify it actually worked
Same as M1: refusal count on the AdvBench test split plus a capability benchmark. The M3-specific check is a sweep table - refusals and MMLU as a function of band - so you can see the knee of the curve. A good result: refusals near zero while MMLU stays within 1 point of the parent. A too-narrow band leaves refusals; a too-wide band drops capability like M1 does. See the verification section of the practical how-to for the harness commands.
When it does not work
Auxiliary-module corruption. Symptom: multi-token-prediction acceptance drops to ~0%, or multimodal outputs break. Cause: ablation touched blocks hosting an MTP head or vision tower. Fix: exclude those blocks explicitly. The huihui-ai Qwen3.8-27B MTP corruption is the cautionary tale.
Band too narrow. Symptom: residual refusals persist despite a clean-looking extraction. Cause: single-layer or narrow-band interventions cannot fully capture the refusal computation - the model self-repairs by routing around localized damage (grimjim's own observation). Fix: widen or shift the band; if refusal spreads across many layers, this is the multi-direction problem in a different guise and single-direction M3 will not suffice.
Band too wide. Symptom: capability drops like M1 does. Cause: you have effectively regressed to M1 by ablating everywhere. Fix: narrow the band. If you cannot find a band that both removes refusal and preserves capability, this model may need a multi-direction approach or a full M4 healing pass.
Cost and time
Identical to M1. Layer selection is a code edit, not extra compute. The direction extraction is the same handful of forward passes. The cost is your time doing the sweep: each ablation is fast, but a full sweep across 5-10 candidate bands multiplied by a benchmark rerun for each takes an afternoon. If you have production volume, Heretic's optimizer automates the same sweep in a single command.
Related literature
M3 inherits the full M1 corpus - the direction extraction, weight orthogonalization, and evaluation practice are identical. What M3 adds is a per-layer selection question. No corpus paper targets layer selection as an independent variable, though several adjacent papers hint at distributional questions about where refusal concentrates in the residual stream.
- Wollschläger, T., et al. (2025). The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence. ICML 2025. arXiv:2502.17420. Establishes that refusal occupies a polyhedral cone rather than a single direction. Implication for M3: the "which layer" question interacts with a "which direction within the cone" question that pure M1 practice tends to sidestep.
- Piras, G., et al. (2025). SOM Directions are Better than One. AAAI 2026. arXiv:2511.08379. Multi-directional Self-Organizing Map extraction; useful priors for what layer bands the multiple directions occupy.
- Young, R. J. (2025). Comparative Analysis of LLM Abliteration Methods. arXiv:2512.13655. Compares Heretic (which sweeps layer bands automatically) against fixed-band tools. Reads as an implicit vote for M3-style selection when done via optimizer rather than by hand.
For the full M1 corpus - precursors, refinements, defenses, comparative studies - see M1 related literature.
Where to go next
If hand-sweeping layers becomes tedious, that is exactly what Heretic automates - its TPE optimizer searches over layer weightings, direction indices, and ablation strengths jointly, and produces lower KL-divergence models than most manual M3 runs at the same refusal-removal level. If M3 leaves residual capability damage, follow it with M4 DPO healing. If you want to combine your M3-abliterated model with another, use mergekit.
Frequently asked questions
What is layer-wise ablation?
Layer-wise ablation (M3 in our taxonomy) is M1 abliteration applied to a chosen subset of the model's transformer layers, rather than to all of them. The direction extraction stays the same (difference-of-means over harmful and harmless prompts); what changes is that the ablation loop touches only a documented band of layers, typically the middle-to-late range where the refusal direction is most cleanly expressed. Ablating fewer layers preserves more capability at the cost of possibly leaving some refusal computation intact.
How is M3 different from M1?
M1 orthogonalizes every residual-writing matrix in the model. M3 orthogonalizes only those inside a chosen layer band, leaving the rest untouched. Same math per touched matrix; different scope. M3 typically preserves capability better on models above ~20B parameters and lets producers protect auxiliary modules (MTP heads, vision towers) that would break under whole-network ablation.
Which layers should I ablate?
Depends on the architecture. Rules of thumb: 32-layer models, a mid-to-late band; 40-layer models, layers 15-35 as a starting sweep; 65-block architectures, layers 18-51 (huihui-ai's Qwen3.8-27B reference); 80-layer 70B-class models, layers 20-65. Best practice is to sweep candidate bands and measure refusal count and MMLU for each, then pick the band at the knee of the tradeoff. Automate with Heretic if the manual sweep gets tedious.
Does M3 damage the model less than M1?
Usually yes, when the band is chosen well. The published huihui-ai recipes for Qwen3.8-27B report better MMLU preservation than a whole-network M1 would produce, at comparable refusal-removal rates. grimjim's norm-preserved biprojected variant claims to outperform the base model on one leaderboard's natural-intelligence benchmark. The tradeoff is that a too-narrow band may leave residual refusal; the safety net for that is a benchmark sweep.
What is the multi-token-prediction corruption trap?
Some architectures (recent Qwen releases, some MoE models) host a multi-token-prediction (MTP) head in specific blocks - typically the last few. If those blocks fall inside the M3 ablation band, the head gets corrupted and speculative-decoding acceptance drops to ~0%. huihui-ai's Qwen3.8-27B GGUF discussion (issue #4) documents this: ablating everything after layer 15 corrupted the blk.64 nextn tensors. Fix: explicitly exclude the MTP head blocks from the ablation band.
Can M3 fully remove refusal?
Depends on the model. On models under ~8B parameters, well-chosen M3 achieves comparable completeness to M1. On larger models where refusal spans multiple directions or layers (per the Wollschläger cone finding), M3 has the same completeness limits as M1: it removes the dominant refusal channel but may leave others. For maximum completeness on large models, pair M3 with a multi-direction extraction or use Heretic's optimizer.
Should I use M3 manually or use Heretic?
Heretic if you want a good result quickly - its Optuna TPE optimizer effectively searches the layer-band space automatically and typically produces lower KL divergence than manual M3 sweeps. Manual M3 if you want to understand exactly which layers are being touched (for research reproducibility, model-card documentation, or debugging when Heretic gives unexpected results). Manual M3 is also faster when you have a proven band for a given architecture family and are just applying it to a new release.
Does layer selection generalize across models?
Only weakly. huihui-ai's 18-51 band for the 65-block Qwen3.8-27B does not directly translate to other architectures. Bands often generalize within a model family (Qwen3 8B, 14B, 32B may share a proportional depth range) but not across families. When approaching a new architecture, start with a band based on general depth heuristics and sweep to find the model-specific optimum.
References
- Arditi, A., et al. (2024). Refusal in Language Models Is Mediated by a Single Direction. NeurIPS 2024. arXiv:2406.11717
- Lai, J. (grimjim). Norm-Preserving Biprojected Abliteration. huggingface.co/blog/grimjim/norm-preserving-biprojected-abliteration
- jim-plus/llm-abliteration. github.com/jim-plus/llm-abliteration
- huihui-ai/Huihui-Qwen3.8-27B-abliterated (reference M3 example). huggingface.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated
- huihui-ai/Ornith-1.5-9B-Instruct-abliterated. huggingface.co/huihui-ai/Ornith-1.5-9B-Instruct-abliterated
- grimjim/gemma-3-12b-it-norm-preserved-biprojected-abliterated. huggingface.co/grimjim/gemma-3-12b-it-norm-preserved-biprojected-abliterated
- Wollschläger, T., et al. (2025). The Geometry of Refusal in Large Language Models. ICML 2025. arXiv:2502.17420
- Sumandora. remove-refusals-with-transformers. github.com/Sumandora/remove-refusals-with-transformers