license: apache-2.0
base_model:
- petruhonk/Qwen3.8-9B-Distill-uncensored-heretic
pipeline_tag: text-generation
library_name: llama.cpp
language: - en
- zh
- es
- fr
- ja
- ko
- ru
tags: - gguf
- llama.cpp
- mtp
- nextn
- speculative-decoding
- imatrix
- q4_k_m
- heretic
- abliterated
- uncensored
- conversational
Qwen3.8-9B-Distill-uncensored-heretic — MTP — Q4_K_M GGUF
Q4_K_M quantization ofpetruhonk/Qwen3.8-9B-Distill-uncensored-heretic,
with the built-in MTP (NextN) head kept in the same file and quantized at
Q8_0 instead of Q4_K so draft acceptance stays high.
- Text trunk: Q4_K_M (
general.file_type = 15, 5.06 BPW class, mixed Q4_K / Q6_K) - MTP block
blk.32.*: Q8_0 for the 8 weight matrices, norms stay F32 - 394 tensors,
qwen35.block_count = 33,qwen35.nextn_predict_layers = 1 - Quantized with a real activation importance matrix (
--nextnincluded) - No separate draft model file needed:
--spec-type draft-mtpreadsblk.32.nextn.*
from this GGUF
| File | Size | Contents |
|---|---|---|
Qwen3.8-9B-Distill-uncensored-heretic-MTP-Q4_K_M.gguf |
5,939,570,176 B | text trunk + MTP head |
Qwen3.8-9B-Distill-uncensored-heretic-MTP-mmproj-f16.gguf |
918,165,952 B | vision projector (F16), for images/video |
SHA256SUMS.txt holds the checksums; calibration.txt, perplexity-corpus.txt andtensor_types.txt are the exact inputs used, so the quantized file is reproducible
byte-for-byte from the BF16 GGUF.
Why the MTP head is quantized differently
MTP is a pure draft layer: the trunk verifies every proposal and discards rejects, so
the head can never change the output distribution — it only decides how many tokens
each verification step accepts. A 4-bit draft head drifts more from the trunk, which
shows up as a lower acceptance rate, which is exactly the throughput you were trying to
buy. Measured on this repo's own weights, 8 CPU cores, greedy, 6 prompts × 300 tokens:
| MTP block precision | Size | Draft acceptance | Mean accepted length | Generation |
|---|---|---|---|---|
| Q8_0 (this repo) | 5.94 GB | 1216 / 1722 = 70.6 % | 3.14 | 9.45 tok/s |
| Q4_K (default Q4_K_M) | 5.83 GB | 1213 / 1731 = 70.1 % | 3.13 | 9.63 tok/s |
Read that table honestly: on this machine the two are within noise of each other
(±0.5 pp acceptance, run-to-run spread 59–89 %), and Q8_0 costs 102 MiB (+1.8 %). The
Q8_0 choice is insurance, not a measured win at this scale — a single 9B-class MTP layer
is barely perturbed by 4-bit quantization, but on smaller draft heads and longer
generations the gap grows, and 102 MiB is cheap. If you need the smallest file, quantizeblk.32.* at Q4_K; nothing else changes.
Quality: how much did Q4_K_M cost?
Perplexity on a held-out multilingual corpus (perplexity-corpus.txt, 4 chunks × 512
ctx, unrelated to the imatrix text), same binary, same flags:
| Model | Perplexity |
|---|---|
| BF16 GGUF (conversion of the source weights) | 5.3983 ± 0.45004 |
| This Q4_K_M | 5.4039 ± 0.44885 |
+0.10 % relative. That is a perplexity number, not a benchmark score — Q4_K_M is lossy,
and a 0.1 % PPL delta does not mean "no quality loss" on reasoning-heavy prompts. What
it does support is that the imatrix-driven recipe did not visibly break the model.
Usage
llama.cpp
llama-server -m Qwen3.8-9B-Distill-uncensored-heretic-MTP-Q4_K_M.gguf \
-c 32768 -np 1 -t 8 --jinja \
--spec-type draft-mtp --spec-draft-n-max 3
--spec-type draft-mtp is required — without it the NextN layer is loaded but never
executed, and you get no speedup. Do not pass -md / a second model: the drafter
lives inside this file. --jinja enables the chat template's tool-calling path.
Images/video:
llama-mtmd-cli -m Qwen3.8-9B-Distill-uncensored-heretic-MTP-Q4_K_M.gguf \
--mmproj Qwen3.8-9B-Distill-uncensored-heretic-MTP-mmproj-f16.gguf \
--image photo.jpg -p "Describe the image."
Ollama
cat > Modelfile <<'MF'
FROM ./Qwen3.8-9B-Distill-uncensored-heretic-MTP-Q4_K_M.gguf
MF
ollama create qwen38-heretic-mtp -f Modelfile
Provenance and credits
Weights, training and the ablation are entirely the work of others; this repository
adds only GGUF conversion + imatrix quantization. Full chain:
- Qwen (Alibaba) —
Qwen/Qwen3.5-9B, the base model family (Apache-2.0). - Empero —
empero-ai/Qwen3.8-9B-Distill,
a full-parameter distillation of Qwen3.8 2.4T-A95B into the Qwen3.5-9B architecture
(~70k curated teacher traces). The 15mtp.*tensors originate here and are
byte-for-byte identical in the repository below — I verified all 15 via HTTP
Range requests (sha256 per tensor), not by trusting the README. - petruhonk —
petruhonk/Qwen3.8-9B-Distill-uncensored-heretic,
the direct source of these weights: a Heretic
ablation (60-trial coarse→mid→fine sweep) that reports 6/100 refusals at KL 0.0306 over
the distillation. The ablation touched onlyattn.o_proj/mlp.down_projin the 32
main layers; the MTP block is untouched base weights, which is why the head still works. - hUh63 — BF16 GGUF conversion with the head folded in as block 32, imatrix
collection with--nextn, Q4_K_M quantization with the MTP block pinned to Q8_0,
and the measurements above.
License: Apache-2.0, inherited from the upstream chain. If you redistribute, keep the
attribution above.
Limitations
- This is a decensored / abliterated community model. Safety alignment has been
deliberately weakened; quantization does not restore it. You are responsible for how
you use it and for any local rules that apply. - Q4_K_M is a lossy format. Norms and biases stay F32 and the MTP block is Q8_0, which
limits the damage, but 4-bit matrix weights are 4-bit matrix weights. - Acceptance rates above were measured on 8 CPU cores with greedy sampling and short
generations. They vary with prompt style (code and structured math accept more than
free-form prose), context length and sampling settings. - Perplexity deltas of this size cannot resolve differences on long-chain reasoning;
benchmark the tasks you care about. - The text trunk is 32 layers; the vision tower lives in the
mmprojfile. Loading the
GGUF without--mmprojgives you a text-only model.