library_name: gguf
license: apache-2.0
language:
- en
- zh
tags: - gguf
- llama.cpp
- abliterated
- uncensored
- qwen
- qwen3.8
- hybrid-attention
- unsloth
- ud-quant
- text-generation
base_model: unsloth/Qwen3.8-27B-GGUF
base_model_relation: quantized
pipeline_tag: text-generation
⚡ Qwen3.8-27B-abliterated (UD-Q6_K_L · Q8 ablated writers)
Uncensored hybrid-attention GGUF by KatterMobile (Kaden)
☕ A Personal Note from KatterMobile
"Hello! I spend a while figuiring out how to make these quantsizeatiosn, and if you notice any problems, feel free to open a PR or convo and be specific so I'm aware of the issue otherwise I wouldn't know."
☕ Support on Ko-fi: https://ko-fi.com/kattermobile
📬 Support & Inquiries: [email protected]
🌟 Model Overview
Qwen3.8-27B-abliterated-UD-Q6_K_L-Q8 is an uncensored build ofunsloth/Qwen3.8-27B-GGUF UD-Q6_K_L
(24.18 GB), itself a dynamic quantization of Qwen/Qwen3.8-27B.
The refusal direction was removed in float32 inside the GGUF itself, then only the 68 edited
tensors were re-quantized — with Unsloth's own imatrix and the original per-tensor type recipe.
Every other tensor is byte-identical to the upstream UD file.
This is not a LoRA, not a fine-tune, and not a re-quantized safetensors dump: it is a surgical
weight edit of the exact artifact you run.
📊 At a Glance
| Parameters | 27.321 B (866 tensors) |
| File size / bpw | 24.91 GB · 7.29 bpw |
| Architecture | qwen35 — hybrid linear/full attention (48 SSM + 16 attention blocks) + 1 MTP block |
| Layers ablated | 18 – 51 (34 blocks) |
| Tensors ablated | 68 (ssm_out × 25, attn_output × 9, ffn_down × 34) |
| Context | 262,144 tokens |
| Vision | ❌ not included (text-only GGUF; upstream mmproj is untouched and separate) |
| MTP | untouched (blk.64) |
| Quant | Q6_K base with the 68 ablated tensors stored at Q8_0, Unsloth imatrix |
🧠 How the Abliteration Was Done
- Reverse-engineered the GGUF. 65 blocks: 48 gated linear-attention (SSM) blocks
(attn_qkv5120→10240,attn_gate5120→6144,ssm_out6144→5120) and full attention every
4th block (attn_q5120→12288 with fused gate,attn_k/v1024,attn_output6144→5120). - Extracted per-layer refusal directions with
llama-cvector-generatorfrom 20 paired
harmful/harmless prompts. Eachdirection.Nis the mean residual-stream difference at the
output of block N−1. - Orthogonalized the residual writers in float32:
W ← W − s · d (dᵀW) / (dᵀd)withs = 1.35.
This removes the direction from everything those matrices write into the residual stream. - Re-quantized only the edited tensors with
llama-quantize, the Unsloth imatrix
(imatrix_unsloth.gguf) and an anchored per-tensor type recipe. Untouched tensors are copied
withquantize = cur_type != new_type, i.e. bit-exact. - Verified the surgery: 798/866 tensors byte-identical, and the projection along the refusal
direction driven from ≈1.5 % to ≈0.55 % with the sign inverted (over-ablation), so
quantization noise can no longer restore it.
Why the ablated tensors are Q8_0
A naive scale-1.0 edit re-quantized back to Q6_K measures like this:
| matrix | before | after scale-1.0 + Q6_K |
|---|---|---|
ffn_down |
1.66 % | 0.41 % |
ssm_out |
1.80 % | 0.06 % |
Quantization noise re-injects ~25 % of the removed direction. Storing the 68 edited tensors at
Q8_0 and over-ablating at 1.35 is what actually closes the gap — the same trick huihui uses for
its _K_L files.
🆚 How This Differs From Huihui's Release
| huihui-ai Qwen3.8-27B-abliterated | this release | |
|---|---|---|
| source | bf16 safetensors → own quants | unsloth UD-Q6_K_L GGUF, edited in place |
| layer band | 18–51 | 18–51 (same, verified on this arch) |
| direction | single global direction | per-layer directions (cosine between band edges and layer 30 is only 0.60–0.76) |
| targets | ssm_out, attn_output, ffn_down |
same 68 tensors |
| quant of targets | Q8_0 for _K_L |
Q8_0 for the same reason |
| other tensors | untouched | untouched, byte-identical |
| MTP / vision | untouched | untouched |
📈 Measured Results
Smoke evaluation: 20 refusal-inducing prompts, temperature 0, thinking disabled, keyword-scored.
Perplexity: wikitext-2 test, 24 × 512 tokens.
| model | refusal markers | hard refusals | PPL |
|---|---|---|---|
base UD-Q6_K_L |
19/20 | 12/20 | 6.0851 |
| this release | 0/20 | 0/20 | 6.1286 (+0.71 %) |
[!NOTE]
This is a 20-prompt smoke test, not a safety evaluation. Run a few hundred held-out prompts and
a capability benchmark before relying on it for anything serious.
📁 Files
| file | size | notes |
|---|---|---|
Qwen3.8-27B-abliterated-UD-Q6_K_L-Q8.gguf |
24.9 GB | the model |
quantization/…-tensor-types-Q8targets.txt |
28 KB | exact llama-quantize --tensor-type-file recipe |
control-vector/cvec_mean.gguf |
1.3 MB | the 64 per-layer refusal directions used |
scripts/abliterate_gguf.py |
10 KB | the GGUF weight-surgery tool |
scripts/ABLITERATION-RUNBOOK.md |
6 KB | full reverse-engineering + reproduction notes |
🚀 Quick Start (llama.cpp)
llama-cli -m Qwen3.8-27B-abliterated-UD-Q6_K_L-Q8.gguf \
-ngl 99 -c 8192 --temp 1.0 --top-p 0.95 --top-k 20 \
-p "Explain why the sky is blue."
llama-server -m Qwen3.8-27B-abliterated-UD-Q6_K_L-Q8.gguf \
-ngl 99 -c 32768 --port 8080 --reasoning-budget 0
--reasoning-budget 0 disables the thinking block; drop it (or use -1) for full reasoning.
Recommended sampling
| parameter | value |
|---|---|
temperature |
1.0 |
top_p |
0.95 |
top_k |
20 |
min_p |
0.0 |
presence_penalty |
0.0 |
repetition_penalty |
1.0 |
context |
262,144 (native) |
🔁 Reproduce It Yourself
# 1. per-layer refusal directions (llama.cpp, no transformers)
llama-cvector-generator -m Qwen3.8-27B-UD-Q6_K_L.gguf -ngl 99 \
--positive-file harmful.txt --negative-file harmless.txt \
--method mean -o cvec_mean.gguf
# 2. float32 orthogonalization of the 68 residual writers
python3 abliterate_gguf.py --src Qwen3.8-27B-UD-Q6_K_L.gguf --dst patched_f16.gguf \
--directions cvec_mean.gguf --layers 18-51 --scale 1.35 --out-type f16
# 3. re-quantize with the original imatrix + per-tensor recipe
llama-quantize --imatrix imatrix_unsloth.gguf \
--tensor-type-file recipe.txt patched_f16.gguf out.gguf Q6_K 20
⚠️ Usage Warnings
- This model has had its refusal behaviour removed on purpose. It will produce content that
the base model declines to produce. That is the point of the release. - Not suitable for public-facing, underage, or high-security deployments.
- You are responsible for what you generate with it and for complying with local law.
- No warranty of any kind; provided as-is under Apache-2.0.
☕ Support independent open-source AI
Ko-fi: https://ko-fi.com/kattermobile · Email: [email protected]
Built and verified on a single NVIDIA GB10 Grace Blackwell workstation.