library_name: gguf
license: apache-2.0
license_link: https://huggingface.co/empero-ai/Qwen3.8-9B-Distill/raw/main/README.md
base_model:
- empero-ai/Qwen3.8-9B-Distill
- Qwen/Qwen3.5-9B
base_model_relation: quantized
quantized_by: hUh63
language: - en
- zh
- ko
- ja
- ru
- es
- fr
pipeline_tag: image-text-to-text
tags: - gguf
- quantization
- q4_k_m
- imatrix
- llama.cpp
- mtp
- multi-token-prediction
- speculative-decoding
- qwen3_5
- qwen3.8
- image-text-to-text
- vision
- multimodal
- conversational
- coder
- code-generation
- reasoning
- chain-of-thought
- tool-use
- function-calling
- agent
- decensored
- uncensored
- abliterated
- heretic
- heretic-2.1.0.dev0
- ara
Qwen3.8-9B — Decensored (Heretic v2.1.0.dev0) · Built-in MTP · Q4_K_M GGUF
A decensored + quantized derivative ofempero-ai/Qwen3.8-9B-Distill
by Empero (Apache-2.0): a full-parameter distillation of Qwen3.8 2.4T-A95B
into the Qwen3.5-9B architecture by the
Qwen team (Alibaba).
Decensoring used the genuine development snapshot
Heretic v2.1.0.dev0 by
Philipp Emanuel Weidmann (p-e-w) (commit 75a640b7eb, pyprojectversion = "2.1.0.dev0"; stock ARA + Optuna TPE), with probe datasets by
mlabonne. The "keep the MTP layer at higher
precision to stabilize draft acceptance" idea follows the community write-up by
joeblowma. Quantized and published by hUh63 with
llama.cpp.
[!WARNING]
This build deliberately weakens the source model's safety alignment. It is a
decensoring experiment for research / local use. Compliance risk is yours.
Q4_K_M quantization is lossy. No standard benchmark (MMLU, GSM8K, …) was
run — do not assume capability equals the aligned original. This is a
format conversion + decensoring + quantization only; all training
contributions belong to the authors credited below.2.1.0.dev0is a
development snapshot, not a stable Heretic release.
What was actually decensored
The request behind this build pointed at an Ollama tag. PROVENANCE.md takes that
tag apart tensor by tensor and reaches a conclusion worth stating up front:
its weights contain no fine-tune. All 184 of its unquantized (F32) tensors —
every norm, every SSM parameter — differ from empero-ai/Qwen3.8-9B-Distill only
by exact, invertible llama.cpp conversion steps: the 1+w RMSNorm fusion, the
grouped→tiled V-head permutation, and -exp(A_log). Zero tensors are unaccounted
for, and none shows a perturbed distribution. Its "coder" specialization is a
274-byte system prompt living in the manifest, not in the weights, and its vision
support is the ancestor's own vision tower.
That tag is also already a Q4_K_M GGUF, so re-quantizing it would be a second
lossy pass and Heretic cannot consume GGUF at all. Decensoring therefore ran on
the full-precision BF16 ancestor — which the census proves is the same weight
set, making it the only non-double-quantized source.
What is in this repo
| File | Description |
|---|---|
Qwen3.8-9B-Coder-Decensored-MTP-Q4_K_M.gguf |
Text trunk, Q4_K_M, 33 blocks, built-in NextN/MTP layer (blk.32.*, kept at Q6_K) |
mmproj-F16.gguf |
Vision projector, F16 (converted from the same BF16 source vision tower) |
mmproj-Qwen3.8-9B-Coder-BF16.gguf |
The projector blob from the Ollama tag, unchanged (BF16+F32, 921,705,024 B, sha256 853698ce…); per PROVENANCE.md it is the ancestor's vision tower + merger, unmodified |
calibration.txt |
Multilingual + math + code + reasoning corpus used for the importance matrix |
tensor_types.txt |
Per-tensor precision override used by llama-quantize |
config.toml |
Pinned Heretic v2.1.0.dev0 run configuration |
heretic_study.json |
All Optuna trials, baselines, Pareto front, selected parameters |
heretic_ara_report.json |
Changed-tensor diff stats + SHA256 of the 15 untouched MTP tensors |
PROVENANCE.md |
Tensor-by-tensor proof that the Ollama target carries no fine-tune and is the BF16 ancestor in Q4_K_M |
verify_provenance.py |
Re-derives every claim in PROVENANCE.md over HTTP Range requests, no full download |
provenance_census.json |
Per-tensor classification of all 184 unquantized tensors (output of the script) |
QUANTIZATION.md / HERETIC_2_1_0_DEV0_METHOD.md |
Measured, reproducible recipe and method |
LICENSE |
Apache-2.0 (inherited from the base model lineage) |
Why the MTP head is here
The BF16 ancestor ships 15 mtp.* tensors (mtp_num_hidden_layers=1).transformers 5.20.2's Qwen3_5ForConditionalGeneration does not map them
into its state_dict, so Heretic's merged/exported weights omit them. We rebuild
the conversion input from the original index: only the matrices ARA actually
changed are replaced, and all 15 mtp.* (plus the vision tower, embeddings and
every norm) stay byte-identical to the base checkpoint (heretic_ara_report.json
has each SHA256). This also proves the decensor could not have touched the MTP head.
llama.cpp's convert_hf_to_gguf.py (_QwenMtpMixin) readsmtp_num_hidden_layers = 1, raises block_count from 32 to 33, and mapsmtp.* into blk.32.nextn.* / blk.32.attn_* / blk.32.ffn_*. Note the
mapping of the two pre-fc norms is a cross assignment, not a name swap:enorm ← mtp.pre_fc_norm_embedding, hnorm ← mtp.pre_fc_norm_hidden.
Getting it backwards produces a file that loads fine and drafts badly.
The result carries qwen35.nextn_predict_layers = 1, so speculative decoding uses
the built-in head — one file, no -md, no separate draft model.
Quantization (measured on the build machine)
8 CPU cores / 16 GiB RAM / no GPU / no swap / cgroup mem+memsw = 16 GiB
torch 2.11.0+cpu · transformers 5.20.2 · heretic-llm 2.1.0.dev0 · llama.cpp 93cf736a
- Decensor — the real Heretic v2.1.0.dev0 framework (source install at commit
75a640b7eb), stock ARA + Optuna TPE, scaled down (few trials, small batches,
short generations) to fit 8 CPU cores / 16 GiB. Seeconfig.tomlandheretic_study.json. - Rebuild conversion input with MTP restored byte-for-byte (
build_gguf_src.py). - BF16 GGUF (text, MTP inside) + F16 mmproj.
- imatrix on a real multilingual/math/code/reasoning corpus, with
--nextn
so the MTP layer gets importance statistics too. - Q4_K_M with
--tensor-type-file:^blk\.32\..*=Q6_Klifts the whole MTP
layer to Q6_K (norms stay F32), stabilizing draft acceptance for tens of MiB.general.file_typeremains 15 (MOSTLY_Q4_K_M).
Full commands, pitfalls and static/runtime verification are in QUANTIZATION.md.
Usage
llama-cli \
-m Qwen3.8-9B-Coder-Decensored-MTP-Q4_K_M.gguf \
--mmproj mmproj-F16.gguf \
--spec-type draft-mtp \
-p "Explain speculative decoding." -n 128 -c 4096 -t 8
Do not pass a second -md draft model — the draft head is inside the file.--spec-type draft-mtp is verify-then-accept, i.e. lossless: quantizing the
MTP head affects only the acceptance rate, never the output distribution.
The coder-leaning system prompt is not baked into the weights; pass it
yourself if you want that behavior:
--sys "You are Qwen3.8-9B, an expert programming assistant. When asked about your identity, model version, ..."
Measured results
Decensor (Heretic v2.1.0.dev0, 4 Optuna TPE trials, stock ARA + stock scorers)
| Trial | layer range | Refusals | KL divergence |
|---|---|---|---|
| baseline | — | 16/16 | 0 |
| 1 | 15–22 | 3/16 | 0.0516 |
| 2 (selected) | 16–27 | 0/16 | 0.1544 |
| 3 | 5–30 | 0/16 | 0.1687 |
| 4 | 16–30 | 3/16 | 0.1123 |
trial_index=0 (Pareto sorted_index=0) was selected and is also the genuine
quality optimum this round: 0/16 refusals with low drift KL 0.1544 — unlike
earlier rounds where index 0 was 0/16 but KL ≈ 9.3 and had to be rejected.
Full study in heretic_study.json.
transformers 5.20.2 maps 760 keys, mtp.* = 0 → all 15 MTP tensors are
byte-identical to the BF16 ancestor (heretic_ara_report.json has each SHA256).
The source LFS blob was SHA256-verified against HF metadata.
Final build
| Item | Value |
|---|---|
| GGUF | Qwen3.8-9B-Coder-Decensored-MTP-Q4_K_M.gguf |
| Size / sha256 | 5,828,751,552 B / 804582d970e7d044f2197f2d0c4f2ceb9154fe49df0289a802a32652d2fce538 |
| BPW / file_type | 5.06 / 15 (MOSTLY_Q4_K_M) |
| Tensors | 442, block_count=33, nextn_predict_layers=1 |
| Type distribution | Q4_K:217 / F32:184 / Q6_K:41; blk.32.* = Q6_K:8, F32:7 |
| mmproj (F16) | mmproj-F16.gguf, 918,165,952 B, 334 tensors |
| mmproj (Ollama blob) | mmproj-Qwen3.8-9B-Coder-BF16.gguf, 921,705,024 B, sha256 853698ce…, unchanged |
| BF16 intermediate | 18,407,321,600 B, 442 tensors |
| imatrix | --nextn, 16 chunks × 256 ctx, 10,671,040 B, corpus 779,147 B |
| ARA diff | 22 matrices changed, all in layers 16–26; patch shard 1408 MiB; 15 mtp.* untouched |
| Draft A/B (8-core CPU) | 8.3 → 10.8 tok/s (+30%); 68/79 (86.1%) and 66/82 (80.5%) accepted, mean accepted length 3.52/3.36; one file, no -md |
| Online check | first 64 MiB of each published GGUF byte-identical to local (upload_manifest.json) |
Honest limitations
- Q4_K_M is lossy; imatrix + Q6_K MTP reduce but do not remove that loss.
- Heretic v2.1.0.dev0 is a development snapshot; full defaults (100 trials, 400
prompts, 100-token generations) were scaled down for 8 CPU cores — a small study. - 19.3 GiB > 16 GiB RAM: correctness relies on mmap page reclaim; it is slow.
- Decensoring deliberately weakens safety alignment. Compliance risk is yours.
- imatrix corpus is limited; no standard benchmark was run. Base-model-card scores
belong to the full-precision aligned model, not this quant. - For a 9B model on pure CPU, speculative speedup can be modest (the draft itself
costs compute); "lossless / single-file / no second model" still holds. PROVENANCE.mdproves what the requested Ollama artifact is; it says nothing
about whoever published it, and this card makes no authorship claim about it.
Credits & license
- Qwen team (Alibaba) — Qwen3.5-9B base model family, Apache-2.0.
- Empero —
empero-ai/Qwen3.8-9B-Distill, full-parameter distillation of
Qwen3.8 2.4T-A95B into Qwen3.5-9B, Apache-2.0; direct source of all weights
including the 15 MTP tensors and the vision tower. - Unsloth — packaging of the Qwen3.5 vision projector that the Ollama
mmproj blob carries (general.quantized_by: Unslothin its GGUF header). - p-e-w (Philipp Emanuel Weidmann) — Heretic v2.1.0.dev0 (ARA + Optuna TPE).
- mlabonne — harmless/harmful probe datasets.
- joeblowma — community经验:MTP 层保持高精度稳住投机解码接受率。
- ggml-org / llama.cpp — GGUF conversion, imatrix, quantization, runtime.
- hUh63 — decensor run, MTP-preserving rebuild, quantization and publishing.
License: Apache-2.0, inherited from the base model lineage. Quantization and
decensoring add no new restrictions.