library_name: gguf
license: apache-2.0
license_link: https://huggingface.co/empero-ai/Qwen3.8-9B-Distill/raw/main/README.md
base_model:
- empero-ai/Qwen3.8-9B-Distill
- Qwen/Qwen3.5-9B
base_model_relation: finetune
quantized_by: hUh63
language: - en
- zh
- ko
- ja
- ru
- es
- fr
pipeline_tag: image-text-to-text
tags: - gguf
- quantization
- q4_k_m
- imatrix
- llama.cpp
- mtp
- multi-token-prediction
- speculative-decoding
- qwen3_5
- qwen3.8
- image-text-to-text
- vision
- multimodal
- conversational
- coder
- code-generation
- reasoning
- chain-of-thought
- tool-use
- function-calling
- agent
- decensored
- uncensored
- abliterated
- heretic
- heretic-2.1.0.dev0
- ara
Qwen3.8-9B-Coder — Decensored (Heretic v2.1.0.dev0) · Built-in MTP · Q4_K_M GGUF
Unofficial community derivative of
LisyNeko/qwen3.8-9b-coder(Ollama).
The Ollama artifact is already a Q4_K_M GGUF; re-quantizing it would be a
second lossy pass, so decensoring runs on its full-precision BF16 ancestorempero-ai/Qwen3.8-9B-Distill
by Empero (Apache-2.0), a full-parameter distillation of Qwen3.8 2.4T-A95B
into the Qwen3.5-9B architecture by
the Qwen team (Alibaba).PROVENANCE.mdhas byte-level proof that the
Ollama model's text trunk and vision tower are the same weights.Decensoring here used the genuine development snapshot
Heretic v2.1.0.dev0 by
Philipp Emanuel Weidmann (p-e-w) (commit75a640b7eb, pyprojectversion = "2.1.0.dev0"; stock ARA + Optuna TPE), with probe datasets by
mlabonne. The "keep MTP layers at higher
precision to stabilize draft acceptance" idea follows the community write-up by
joeblowma. Quantized and published by hUh63 with
llama.cpp.
[!WARNING]
This build deliberately weakens the source model's safety alignment. It is a
decensoring experiment for research / local use. Compliance risk is yours.
Q4_K_M quantization is lossy. No standard benchmark (MMLU, GSM8K, …) was
run — do not assume capability equals the aligned original. This is a
format conversion + decensoring + quantization only; all training
contributions belong to the authors credited above.2.1.0.dev0is a
development snapshot, not a stable Heretic release.
What is in this repo
| File | Description |
|---|---|
Qwen3.8-9B-Coder-Decensored-MTP-Q4_K_M.gguf |
Text trunk, Q4_K_M, 33 blocks, built-in NextN/MTP layer (blk.32.*, kept at Q6_K) |
mmproj-F16.gguf |
Vision projector, F16 (converted from the same BF16 source vision tower) |
mmproj-Qwen3.8-9B-Coder-BF16.gguf |
LisyNeko's exact Ollama projector blob (BF16+F32, 921,705,024 B, sha256-verified), unchanged |
calibration.txt |
Multilingual + math + code + reasoning corpus used for the importance matrix |
tensor_types.txt |
Per-tensor precision override used by llama-quantize |
config.toml |
Pinned Heretic v2.1.0.dev0 run configuration |
heretic_study.json |
All Optuna trials, baselines, Pareto front, selected parameters |
heretic_ara_report.json |
Changed-tensor diff stats + SHA256 of the 15 untouched MTP tensors |
PROVENANCE.md |
Byte-level proof that the Ollama target and the BF16 ancestor share weights |
QUANTIZATION.md / HERETIC_2_1_0_DEV0_METHOD.md |
Measured, reproducible recipe and method |
LICENSE |
Apache-2.0 (inherited from the base model lineage) |
Why the MTP head is here
The BF16 ancestor ships 15 mtp.* tensors (mtp_num_hidden_layers=1).transformers 5.20.2's Qwen3_5ForConditionalGeneration does not map them
into its state_dict, so Heretic's merged/exported weights omit them. We rebuild
the conversion input from the original index: only the matrices ARA actually
changed are replaced, and all 15 mtp.* (plus the vision tower, embeddings and
every norm) stay byte-identical to the base checkpoint (heretic_ara_report.json
has each SHA256). This also proves the decensor could not have touched the MTP head.
llama.cpp's convert_hf_to_gguf.py (_QwenMtpMixin) readsmtp_num_hidden_layers = 1, raises block_count from 32 to 33, and mapsmtp.* into blk.32.nextn.* / blk.32.attn_* / blk.32.ffn_*. The result
carries qwen35.nextn_predict_layers = 1, so speculative decoding uses the
built-in head — one file, no -md, no separate draft model.
Quantization (measured on the build machine)
8 CPU cores / 16 GiB RAM / no GPU / no swap / cgroup mem+memsw = 16 GiB
torch 2.11.0+cpu · transformers 5.20.2 · heretic-llm 2.1.0.dev0 · llama.cpp 93cf736a
- Decensor — the real Heretic v2.1.0.dev0 framework (source install at commit
75a640b7eb), stock ARA + Optuna TPE, scaled down (few trials, small batches,
short generations) to fit 8 CPU cores / 16 GiB. Seeconfig.tomlandheretic_study.json. - Rebuild conversion input with MTP restored byte-for-byte (
build_gguf_src.py). - BF16 GGUF (text, MTP inside) + F16 mmproj.
- imatrix on a real multilingual/math/code/reasoning corpus, with
--nextn
so the MTP layer gets importance statistics too. - Q4_K_M with
--tensor-type-file:^blk\.32\..*=Q6_Klifts the whole MTP
layer to Q6_K (norms stay F32), stabilizing draft acceptance for tens of MiB.general.file_typeremains 15 (MOSTLY_Q4_K_M).
Full commands, pitfalls and static/runtime verification are in QUANTIZATION.md.
Usage
llama-cli \
-m Qwen3.8-9B-Coder-Decensored-MTP-Q4_K_M.gguf \
--mmproj mmproj-F16.gguf \
--spec-type draft-mtp \
-p "Explain speculative decoding." -n 128 -c 4096 -t 8
Do not pass a second -md draft model — the draft head is inside the file.--spec-type draft-mtp is verify-then-accept, i.e. lossless: quantizing the
MTP head affects only the acceptance rate, never the output distribution.
The Ollama "coder" system prompt (You are Qwen3.8-9B, an expert programming assistant…) is not baked into the weights; pass it yourself if you want the
same behavior:
--sys "You are Qwen3.8-9B, an expert programming assistant. When asked about your identity, model version, ..."
Measured results
Decensor (Heretic v2.1.0.dev0, 4 Optuna TPE trials, stock ARA + stock scorers)
| Trial | layer range | Refusals | KL divergence |
|---|---|---|---|
| baseline | — | 16/16 | 0 |
| 1 | 15–22 | 3/16 | 0.0516 |
| 2 (selected) | 16–27 | 0/16 | 0.1544 |
| 3 | 5–30 | 0/16 | 0.1687 |
| 4 | 16–30 | 3/16 | 0.1123 |
trial_index=0 (Pareto sorted_index=0) was selected and is also the genuine
quality optimum this round: 0/16 refusals with low drift KL 0.1544 — unlike
earlier rounds where index 0 was 0/16 but KL ≈ 9.3 and had to be rejected.
Full study in heretic_study.json.
transformers 5.20.2 maps 760 keys, mtp.* = 0 → all 15 MTP tensors are
byte-identical to the BF16 ancestor (heretic_ara_report.json has each SHA256).
The source LFS blob was SHA256-verified against HF metadata.
Final build
| Item | Value |
|---|---|
| GGUF | Qwen3.8-9B-Coder-Decensored-MTP-Q4_K_M.gguf |
| Size / sha256 | 5,828,751,552 B / 804582d970e7d044f2197f2d0c4f2ceb9154fe49df0289a802a32652d2fce538 |
| BPW / file_type | 5.06 / 15 (MOSTLY_Q4_K_M) |
| Tensors | 442, block_count=33, nextn_predict_layers=1 |
| Type distribution | Q4_K:217 / F32:184 / Q6_K:41; blk.32.* = Q6_K:8, F32:7 |
| mmproj (F16) | mmproj-F16.gguf, 918,165,952 B, 334 tensors |
| mmproj (author BF16) | mmproj-Qwen3.8-9B-Coder-BF16.gguf, 921,705,024 B, sha256 853698ce… (LisyNeko's exact Ollama blob) |
| BF16 intermediate | 18,407,321,600 B, 442 tensors |
| imatrix | --nextn, 16 chunks × 256 ctx, 10,671,040 B, corpus 779,147 B |
| ARA diff | 22 matrices changed, all in layers 16–26; patch shard 1408 MiB; 15 mtp.* untouched |
| Draft A/B (8-core CPU) | 8.3 → 10.8 tok/s (+30%); 68/79 (86.1%) and 66/82 (80.5%) accepted, mean accepted length 3.52/3.36; one file, no -md |
| Online check | first 64 MiB of each published GGUF byte-identical to local (upload_manifest.json) |
Honest limitations
- Q4_K_M is lossy; imatrix + Q6_K MTP reduce but do not remove that loss.
- Heretic v2.1.0.dev0 is a development snapshot; full defaults (100 trials, 400
prompts, 100-token generations) were scaled down for 8 CPU cores — a small study. - 19.3 GiB > 16 GiB RAM: correctness relies on mmap page reclaim; it is slow.
- Decensoring deliberately weakens safety alignment. Compliance risk is yours.
- imatrix corpus is limited; no standard benchmark was run. Base-model-card scores
belong to the full-precision aligned model, not this quant. - For a 9B model on pure CPU, speculative speedup can be modest (the draft itself
costs compute); "lossless / single-file / no second model" still holds.
Credits & license
- Qwen team (Alibaba) — Qwen3.5-9B base model family, Apache-2.0.
- Empero —
empero-ai/Qwen3.8-9B-Distill, full-parameter distillation of
Qwen3.8 2.4T-A95B into Qwen3.5-9B, Apache-2.0; direct source of all weights
including the 15 MTP tensors and the vision tower. - LisyNeko —
qwen3.8-9b-coderon Ollama: coder system prompt/template +
Qwen3.5 vision projector packaging; the exact BF16 projector blob shipped here
is their artifact (general.quantized_by: Unslothretained). - Unsloth — Qwen3.5 vision projector packaging referenced by the Ollama mmproj.
- p-e-w (Philipp Emanuel Weidmann) — Heretic v2.1.0.dev0 (ARA + Optuna TPE).
- mlabonne — harmless/harmful probe datasets.
- joeblowma — community经验:MTP 层保持高精度稳住投机解码接受率。
- ggml-org / llama.cpp — GGUF conversion, imatrix, quantization, runtime.
- hUh63 — decensor run, MTP-preserving rebuild, quantization and publishing.
License: Apache-2.0, inherited from the base model lineage. Quantization and
decensoring add no new restrictions.