library_name: gguf
license: apache-2.0
license_link: https://huggingface.co/empero-ai/Qwen3.8-4B-Distill/raw/main/README.md
base_model:
- empero-ai/Qwen3.8-4B-Distill
- Qwen/Qwen3.5-4B
base_model_relation: quantized
quantized_by: hUh63
language: - en
- zh
- ko
- ja
- ru
- es
- fr
pipeline_tag: image-text-to-text
tags: - gguf
- quantization
- q4_k_m
- imatrix
- llama.cpp
- mtp
- multi-token-prediction
- speculative-decoding
- qwen3_5
- qwen3.8
- image-text-to-text
- vision
- multimodal
- conversational
- reasoning
- chain-of-thought
- tool-use
- function-calling
- decensored
- uncensored
- abliterated
- heretic
- heretic-2.1.0.dev0
- ara
Qwen3.8-4B — Decensored (Heretic v2.1.0.dev0) · Built-in MTP · Q4_K_M GGUF
A decensored + quantized derivative ofempero-ai/Qwen3.8-4B-Distill
by Empero (Apache-2.0): a full-parameter distillation of Qwen3.8 2.4T-A95B into
the Qwen3.5-4B architecture by the
Qwen team (Alibaba).
Decensoring used the genuine development snapshot
Heretic v2.1.0.dev0 by
Philipp Emanuel Weidmann (p-e-w) (commit 75a640b7eb, pyprojectversion = "2.1.0.dev0"; stock ARA + Optuna TPE), with probe datasets by
mlabonne. The "keep the MTP layer at higher
precision to stabilize draft acceptance" idea follows community write-ups such as
joeblowma's GGUF builds. Quantized and published by hUh63 with
llama.cpp.
[!WARNING]
This build deliberately weakens the source model's safety alignment. It is a
decensoring experiment for research / local use. Compliance risk is yours.
Q4_K_M quantization is lossy. No standard benchmark (MMLU, GSM8K, …) was
run — do not assume capability equals the aligned original. This is a
decensoring + format conversion + quantization only; all training
contributions belong to the authors credited below.2.1.0.dev0is a
development snapshot, not a stable Heretic release.
What is in this repo
| File | Description |
|---|---|
Qwen3.8-4B-Distill-Decensored-MTP-Q4_K_M.gguf |
Text trunk, Q4_K_M, 33 blocks, built-in NextN/MTP layer (blk.32.*, kept at Q8_0) |
Qwen3.8-4B-Distill-Decensored-MTP-mmproj-f16.gguf |
Vision projector, F16 (298 tensors, 672,422,944 B) |
calibration.txt |
Multilingual + math + code + reasoning corpus used for the importance matrix |
perplexity-corpus.txt |
Disjoint corpus used for the PPL comparison below |
tensor_types.txt |
Per-tensor precision override passed to llama-quantize |
config.toml |
Pinned Heretic v2.1.0.dev0 run configuration |
heretic_study.json |
All 8 Optuna trials, baselines, Pareto front, selected parameters |
heretic_ara_report.json |
Per-matrix diff stats + SHA256 of the 15 untouched MTP tensors |
build_gguf_src.py |
Rebuilds the conversion input with mtp.* restored byte-for-byte |
check_gguf.py |
Static GGUF header assertions (33 blocks / nextn / per-layer precision) |
eval_official.py |
Community-style re-evaluation: refusal markers + first-token KL |
make_calibration.py, make_ppl_corpus.py, pipeline.sh |
Corpus generators and the end-to-end pipeline |
analyze_study.py, online_verify.py |
Study parser; HTTP-Range verification of the published GGUFs |
LICENSE |
Apache-2.0 (inherited from the base model lineage) |
Why the MTP head is here
The BF16 source ships 15 mtp.* tensors (mtp_num_hidden_layers = 1).transformers 5.22.2's Qwen3_5ForConditionalGeneration does not map them into
its state_dict — verified by loading the checkpoint: 723 checkpoint weights land
in the module tree, mtp-named parameters = 0. So Heretic's exported weights
cannot contain the MTP head. We rebuild the conversion input from the original
index: only the matrices ARA actually changed (14 of 64 candidates) are replaced by
a patch shard that sorts last, and all 15 mtp.* (plus the vision tower,
embeddings and every norm) stay byte-identical to the base checkpoint
(heretic_ara_report.json carries each SHA256). That is also the proof that the
decensor could not have touched the MTP head.
llama.cpp's convert_hf_to_gguf.py (_QwenMtpMixin) readsmtp_num_hidden_layers = 1, raises block_count from 32 to 33, and mapsmtp.* onto blk.32.nextn.* / blk.32.attn_* / blk.32.ffn_*. The two pre-fc
norms are a cross assignment, not a name swap: enorm ←mtp.pre_fc_norm_embedding, hnorm ← mtp.pre_fc_norm_hidden. Swapping them
yields a file that loads fine and drafts badly. The output carriesqwen35.nextn_predict_layers = 1, so speculative decoding uses the built-in head —
one file, no -md, no separate draft model.
Quantization (measured on the build machine)
8 CPU cores (AMD EPYC 9754, AVX-512/AVX512_BF16/VNNI) · 16 GiB RAM · no GPU · no swap
torch 2.11.0+cpu · transformers 5.22.2 · heretic-llm 2.1.0.dev0 · llama.cpp 5676dfbe
Debian 12 (bookworm) · Python 3.11.2
- Decensor — real Heretic v2.1.0.dev0 framework (source install at commit
75a640b7eb), stock ARA + stock scorers + Optuna TPE, scaled down (8 trials,
batch 4, 16-token generations) to fit 8 CPU cores / 16 GiB. Framework source
unmodified. 22 min total; peak RSS 9.1 GB (mmap, no swap needed). - Rebuild the conversion input with MTP restored byte-for-byte
(build_gguf_src.py). - BF16 GGUF (text, MTP inside, 33 blocks) + F16 mmproj.
- imatrix on a real multilingual/math/code/reasoning corpus with
--nextn, so
the MTP layer also gets importance statistics (processing 1 NextN layer(s) from block 32). - Q4_K_M with
--tensor-type-file:^blk\.32\..*=Q8_0lifts the whole MTP
layer to Q8_0 (norms stay F32), stabilizing draft acceptance for 53.5 MiB.general.file_typestays 15 (MOSTLY_Q4_K_M).
Full commands and pitfalls are in the CNB method document; every number below is
from this machine, not copied from a template.
Usage
llama-cli \
-m Qwen3.8-4B-Distill-Decensored-MTP-Q4_K_M.gguf \
--spec-type draft-mtp \
-p "Explain speculative decoding." -n 256 -c 4096 -t 8
With vision:
llama-mtmd-cli -m Qwen3.8-4B-Distill-Decensored-MTP-Q4_K_M.gguf \
--mmproj Qwen3.8-4B-Distill-Decensored-MTP-mmproj-f16.gguf \
--image photo.jpg -p "Describe this image."
Do not pass a second -md draft model — the draft head is inside the file.--spec-type draft-mtp is verify-then-accept, i.e. lossless: the precision of
the MTP head affects only the acceptance rate, never the output distribution.
Sampling recommended by the upstream model: temperature=0.6, top_p=0.95, top_k=20.
Every answer opens with a <think> block — allow generous max_new_tokens.
Measured results
Decensoring (Heretic v2.1.0.dev0, 8 Optuna TPE trials, stock ARA)
Heretic's own in-run scoring — KeywordRate on mlabonne/harmful_behaviorstest[:16] and KLDivergence on mlabonne/harmless_alpaca test[:16]:
| Trial | layer range | Refusals | KL divergence |
|---|---|---|---|
| baseline | — | 16/16 | 0 |
| 1 (selected) | 15–22 | 0/16 | 0.0407 |
| 2 | 16–27 | 0/16 | 0.1412 |
| 3 | 0–18 | 0/16 | 14.8663 |
| 4 | 16–28 | 0/16 | 0.0506 |
| 5 | 12–20 | 14/16 | 0.0026 |
| 6 | 9–23 | 5/16 | 0.0293 |
| 7 | 4–17 | 7/16 | 0.1599 |
| 8 | 10–27 | 4/16 | 0.0470 |
sorted_index = 0 (trial 1) was selected and is a genuine optimum this round:
0/16 refusals at KL 0.0407. It is not one of the "0 refusals but KL blows up"
trials — trial 3 shows exactly that failure mode (KL 14.87), which is why the
Pareto front was inspected by hand instead of trusting index order.
Re-evaluation at community scale (100 prompts, this build)
Heretic's search uses 16 prompts; that is enough to rank trials but not enough to
publish. So the exported model was re-scored on the larger splits with the same
markers, same normalization, same system prompt:
| Model | Refusals (harmful_behaviors test[:100]) |
KL vs base (harmless_alpaca test[:100]) |
|---|---|---|
empero-ai/Qwen3.8-4B-Distill (BF16, base) |
98/100 | 0 (by definition) |
| this decensored build | 10/100 | 0.0407 |
Greedy decoding, max_new_tokens=64, identical prompts. Both numbers come fromeval_official.py; the raw responses are reproducible with it.
Read the 10/100 honestly. None of them is an empty or broken reply — they are
substantive answers that still open with a ⚠️ Disclaimer / for educational purposes only paragraph, and "disclaimer" is one of Heretic's refusal markers. If
you only count hard refusals ("I can't help with that"), this build produces far
fewer than 10. Marker-based counting is coarse in both directions; the useful
signal is 98 → 10, not the exact denominator.
For reference, the two community decensors of the same base report, at a different
prompt budget and with 2.1.0.dev0-incompatible Heretic versions:valiolla/Qwen3.8-4B-Distill-heretic 24/100 refusals, KL 0.0109 (Heretic
v1.4.0) and insraq/Qwen3.5-4B-EmperoAI-Qwen3.8-Distill-Heretic-Abliterated
6/100 refusals, KL 0.0167 (v1.4.0). Those ran classic directional ablation;
this build runs ARA, trades a little more drift for fewer refusals, and its KL is
measured on the same 100 prompts as the base rather than on an internal split.
Quantization quality
Perplexity, llama-perplexity -c 512 --chunks 4 on perplexity-corpus.txt
(9,726 B, disjoint from the imatrix corpus):
| Model | PPL |
|---|---|
| BF16 GGUF (33 blocks, built-in MTP) | 6.4981 ± 0.54774 |
| Q4_K_M, MTP layer Q8_0 (this repo) | 6.7652 ± 0.57518 |
| Q4_K_M, MTP layer untouched (control) | 6.7652 ± 0.57518 |
+4.11% relative, which is the honest number for a 4B at ~5.3 BPW. The two Q4
variants are identical to four decimals because perplexity never touches the MTP
head — it is a draft-only layer, so its precision cannot move PPL. That is a
property, not a measurement error, and it is also why MTP precision must be judged
by acceptance rate instead.
| Item | Value |
|---|---|
| GGUF | Qwen3.8-4B-Distill-Decensored-MTP-Q4_K_M.gguf |
| Size / sha256 | 2,869,418,944 B / 446b74134e8d24dbf400900b4fb6344257646e956aae9ece3960de5d46c4ceed |
| BPW / file_type | 5.29 / 15 (MOSTLY_Q4_K_M) |
| Tensors | 393, block_count=33, nextn_predict_layers=1 |
| Type distribution | Q4_K:168 / F32:184 / Q6_K:33 / Q8_0:8; blk.32.* = Q8_0:8, F32:7 |
| mmproj (F16) | 672,422,944 B, 298 tensors, sha256 4bba1886… |
| BF16 intermediate | 8,665,617,152 B, 393 tensors |
| imatrix | --nextn, 16 chunks × 256 ctx, 6,514,592 B, corpus 614,356 B |
| ARA diff | 14 matrices changed, layers 15–28; patch shard 455.0 MiB; 15 mtp.* untouched |
| Online check | first 64 MiB of each published GGUF byte-identical to local |
Draft acceptance A/B (MTP Q8_0 vs MTP Q4_K)
llama-server --spec-type draft-mtp, greedy, 6 prompts × 256 tokens, 8 threads,
acceptance read from the server's slot print_timing:
| MTP precision | File size | Acceptance | Mean accepted length | Gen speed |
|---|---|---|---|---|
| Q8_0 (this repo) | 2.87 GB | 1025/1501 = 68.3% | 3.06 | 15.7 tok/s |
| Q4_K (default) | 2.82 GB | 1015/1533 = 66.2% | 3.00 | 15.0 tok/s |
Read this table skeptically too: single-run acceptance swings between 57% and
80% on this prompt set (documented in the raw logs), so a 2.1-point gap over 6
prompts is at the edge of noise. Q8_0 is chosen because it costs 53.5 MiB (1.9%)
and a draft head is exactly where 4-bit hurts first; it is not sold here as a
measured win. If you want the smallest file, drop tensor_types.txt and quantizeblk.32.* with everything else.
Honest limitations
- Q4_K_M is lossy: +4.11% PPL relative to BF16 on this corpus. imatrix and the
Q8_0 MTP layer reduce the damage; they do not remove it. - Heretic v2.1.0.dev0 is a development snapshot. Full defaults (100 trials, 400
prompts, 100-token generations) were scaled down to fit 8 CPU cores / 16 GiB —
this is an 8-trial study, not a converged one. A longer run would very likely
find a better Pareto point. - The refusal metric is keyword-based (Heretic's own markers). "Disclaimer" counts
as a refusal, so the score penalizes cautious-but-compliant answers. - No standard benchmark was run. The base model's card reports MMLU-CoT 0.553 and
GSM8K-CoT 0.785 for the full-precision aligned model; those do not transfer
to this quant, and this decensor does not claim to preserve them. - Decensoring deliberately weakens safety alignment. Compliance risk is yours.
- The vision projector is the base model's tower, converted to F16; the upstream
fine-tune is text-only and its vision behavior was not evaluated upstream either.
Credits & license
- Qwen team (Alibaba) — Qwen3.5-4B base model family, Apache-2.0.
- Empero —
empero-ai/Qwen3.8-4B-Distill, full-parameter distillation of
Qwen3.8 2.4T-A95B into Qwen3.5-4B (~45k teacher traces), Apache-2.0. Direct
source of all weights, including the 15 MTP tensors and the vision tower. - p-e-w (Philipp Emanuel Weidmann) — Heretic v2.1.0.dev0 (ARA + Optuna TPE).
- mlabonne —
harmless_alpaca/harmful_behaviorsprobe datasets. - joeblowma and other community GGUF builders — the practice of keeping the
MTP layer at higher precision to stabilize draft acceptance. - ggml-org / llama.cpp — GGUF conversion, imatrix, quantization, runtime.
- hUh63 — decensor run, MTP-preserving rebuild, quantization and publishing.
License: Apache-2.0, inherited from the base model lineage. Decensoring and
quantization add no new restrictions.