← back to catalog · registered 2026-11-13 10:03

hUh63/Qwen3.8-4B-Distill-Decensored-MTP-Q4_K_M-GGUF

hUh63 4B GGUF multimodal
Your rig guess connected
? Why do I need an app?
Reading your rig…

This is a rough estimate. Install the free app - we'll show exact numbers.

Reading real hardware from your app right now. Numbers below are exact.

Below is the per-quantization compatibility for this model.

curl -H "Authorization: Bearer $ABL_KEY" \
     "https://abliteration.org/api/v1/models/hUh63%2FQwen3.8-4B-Distill-Decensored-MTP-Q4_K_M-GGUF"
Response includes
  • classification m8
  • files 22
  • author_summary 7 models
  • readme_text full
10 credits · hourly refresh · ~4 KB payload Get an API key →
Abliteration classifier · v1.0.0
M8
Primary method

Repackaging (quantization)

Applied on top of direct removal inherited from the base model.
Confidence
MEDIUM
Inherited from base model
Why this label 3 signals
Method inferred from partial signals - repository name, related files, or tag patterns. Producer identity not confirmed; label may sharpen or shift as we gather more evidence.
  • 'abliterated' in name/tags
  • is_gguf=1
  • assume M1 (base ablation) + M8 (GGUF quant) - default when producer unknown
Refusal direction extraction

No specific extraction method could be identified for this model. The producer either did not document it or used a proprietary pipeline.

What is a refusal direction? →
Downloads · 30-day
328
Likes
0
Model age
1w ago
created 2026-11-13
Downloads over time
Now 328 → from 0 ↑ 0%
0120241361 0 on Nov 13 328 on Nov 22 Nov
Nov 13 → Nov 22 · 10 snapshots · spans 9 days

Genealogy 0 direct forks

Full fork graph →

This model's place in the market. Above: what it was derived from. Below: the tree of everything derived from it.

Metadata

License
apache-2.0
Languages
en zh ko ja ru es fr
Quantizations
Q4_K
Tags
gguf quantization q4_k_m imatrix llama.cpp mtp multi-token-prediction speculative-decoding qwen3_5 qwen3.8 image-text-to-text vision

Related

Total size
2.67 GB
Files
22
Quantizations
3
Registered
2026-11-13 10:03
Last updated on HF
2026-11-14 13:47

Files by quantization

Q4_K 1 file 2.67 GB
Qwen3.8-4B-Distill-Decensored-MTP-Q4_K_M.gguf 2.67 GB 446b7413 download
F16 1 file 641 MB
Qwen3.8-4B-Distill-Decensored-MTP-mmproj-f16.gguf 641 MB 4bba1886 download
Auxiliary files 20 files 706 KB
calibration.txt 600 KB 711fe1aa download
README.md 13.6 KB 5135739c download
make_calibration.py 13.4 KB e55882ca download
LICENSE 11.1 KB d6456956 download
heretic_study.json 10.7 KB d6188bfd download
make_ppl_corpus.py 10.3 KB 007b0304 download
perplexity-corpus.txt 9.50 KB 10dcb200 download
build_gguf_src.py 6.56 KB 22432dc5 download
heretic_ara_report.json 5.93 KB 25ec03d9 download
eval_official.py 4.90 KB e6525fa0 download
check_gguf.py 4.68 KB 1ab1efd3 download
analyze_study.py 4.58 KB 78c4b32a download
config.toml 2.86 KB 0552e7e7 download
online_verify.py 1.99 KB 96ef8980 download
pipeline.sh 1.94 KB 80329258 download
.gitattributes 1.65 KB b3120d50 download
online_verify.json 1.08 KB 0d596389 download
upload_manifest.json 479 B 35e9e0bd download
SHA256SUMS.txt 403 B fed4c126 download
tensor_types.txt 18.0 B ff96bc3c download

README current version from Hugging Face


library_name: gguf
license: apache-2.0
license_link: https://huggingface.co/empero-ai/Qwen3.8-4B-Distill/raw/main/README.md
base_model:

  • empero-ai/Qwen3.8-4B-Distill
  • Qwen/Qwen3.5-4B
    base_model_relation: quantized
    quantized_by: hUh63
    language:
  • en
  • zh
  • ko
  • ja
  • ru
  • es
  • fr
    pipeline_tag: image-text-to-text
    tags:
  • gguf
  • quantization
  • q4_k_m
  • imatrix
  • llama.cpp
  • mtp
  • multi-token-prediction
  • speculative-decoding
  • qwen3_5
  • qwen3.8
  • image-text-to-text
  • vision
  • multimodal
  • conversational
  • reasoning
  • chain-of-thought
  • tool-use
  • function-calling
  • decensored
  • uncensored
  • abliterated
  • heretic
  • heretic-2.1.0.dev0
  • ara

Qwen3.8-4B — Decensored (Heretic v2.1.0.dev0) · Built-in MTP · Q4_K_M GGUF

A decensored + quantized derivative of
empero-ai/Qwen3.8-4B-Distill
by Empero (Apache-2.0): a full-parameter distillation of Qwen3.8 2.4T-A95B into
the Qwen3.5-4B architecture by the
Qwen team (Alibaba).

Decensoring used the genuine development snapshot
Heretic v2.1.0.dev0 by
Philipp Emanuel Weidmann (p-e-w) (commit 75a640b7eb, pyproject
version = "2.1.0.dev0"; stock ARA + Optuna TPE), with probe datasets by
mlabonne. The "keep the MTP layer at higher
precision to stabilize draft acceptance" idea follows community write-ups such as
joeblowma's GGUF builds. Quantized and published by hUh63 with
llama.cpp.

[!WARNING]
This build deliberately weakens the source model's safety alignment. It is a
decensoring experiment for research / local use. Compliance risk is yours.
Q4_K_M quantization is lossy. No standard benchmark (MMLU, GSM8K, …) was
run
— do not assume capability equals the aligned original. This is a
decensoring + format conversion + quantization only; all training
contributions belong to the authors credited below. 2.1.0.dev0 is a
development snapshot, not a stable Heretic release.

What is in this repo

File Description
Qwen3.8-4B-Distill-Decensored-MTP-Q4_K_M.gguf Text trunk, Q4_K_M, 33 blocks, built-in NextN/MTP layer (blk.32.*, kept at Q8_0)
Qwen3.8-4B-Distill-Decensored-MTP-mmproj-f16.gguf Vision projector, F16 (298 tensors, 672,422,944 B)
calibration.txt Multilingual + math + code + reasoning corpus used for the importance matrix
perplexity-corpus.txt Disjoint corpus used for the PPL comparison below
tensor_types.txt Per-tensor precision override passed to llama-quantize
config.toml Pinned Heretic v2.1.0.dev0 run configuration
heretic_study.json All 8 Optuna trials, baselines, Pareto front, selected parameters
heretic_ara_report.json Per-matrix diff stats + SHA256 of the 15 untouched MTP tensors
build_gguf_src.py Rebuilds the conversion input with mtp.* restored byte-for-byte
check_gguf.py Static GGUF header assertions (33 blocks / nextn / per-layer precision)
eval_official.py Community-style re-evaluation: refusal markers + first-token KL
make_calibration.py, make_ppl_corpus.py, pipeline.sh Corpus generators and the end-to-end pipeline
analyze_study.py, online_verify.py Study parser; HTTP-Range verification of the published GGUFs
LICENSE Apache-2.0 (inherited from the base model lineage)

Why the MTP head is here

The BF16 source ships 15 mtp.* tensors (mtp_num_hidden_layers = 1).
transformers 5.22.2's Qwen3_5ForConditionalGeneration does not map them into
its state_dict — verified by loading the checkpoint: 723 checkpoint weights land
in the module tree, mtp-named parameters = 0. So Heretic's exported weights
cannot contain the MTP head. We rebuild the conversion input from the original
index: only the matrices ARA actually changed (14 of 64 candidates) are replaced by
a patch shard that sorts last, and all 15 mtp.* (plus the vision tower,
embeddings and every norm) stay byte-identical to the base checkpoint
(heretic_ara_report.json carries each SHA256). That is also the proof that the
decensor could not have touched the MTP head.

llama.cpp's convert_hf_to_gguf.py (_QwenMtpMixin) reads
mtp_num_hidden_layers = 1, raises block_count from 32 to 33, and maps
mtp.* onto blk.32.nextn.* / blk.32.attn_* / blk.32.ffn_*. The two pre-fc
norms are a cross assignment, not a name swap: enorm ←
mtp.pre_fc_norm_embedding, hnorm ← mtp.pre_fc_norm_hidden. Swapping them
yields a file that loads fine and drafts badly. The output carries
qwen35.nextn_predict_layers = 1, so speculative decoding uses the built-in head —
one file, no -md, no separate draft model.

Quantization (measured on the build machine)

8 CPU cores (AMD EPYC 9754, AVX-512/AVX512_BF16/VNNI) · 16 GiB RAM · no GPU · no swap
torch 2.11.0+cpu · transformers 5.22.2 · heretic-llm 2.1.0.dev0 · llama.cpp 5676dfbe
Debian 12 (bookworm) · Python 3.11.2
  1. Decensor — real Heretic v2.1.0.dev0 framework (source install at commit
    75a640b7eb), stock ARA + stock scorers + Optuna TPE, scaled down (8 trials,
    batch 4, 16-token generations) to fit 8 CPU cores / 16 GiB. Framework source
    unmodified. 22 min total; peak RSS 9.1 GB (mmap, no swap needed).
  2. Rebuild the conversion input with MTP restored byte-for-byte
    (build_gguf_src.py).
  3. BF16 GGUF (text, MTP inside, 33 blocks) + F16 mmproj.
  4. imatrix on a real multilingual/math/code/reasoning corpus with --nextn, so
    the MTP layer also gets importance statistics (processing 1 NextN layer(s) from block 32).
  5. Q4_K_M with --tensor-type-file: ^blk\.32\..*=Q8_0 lifts the whole MTP
    layer to Q8_0 (norms stay F32), stabilizing draft acceptance for 53.5 MiB.
    general.file_type stays 15 (MOSTLY_Q4_K_M).

Full commands and pitfalls are in the CNB method document; every number below is
from this machine, not copied from a template.

Usage

llama-cli \
  -m Qwen3.8-4B-Distill-Decensored-MTP-Q4_K_M.gguf \
  --spec-type draft-mtp \
  -p "Explain speculative decoding." -n 256 -c 4096 -t 8

With vision:

llama-mtmd-cli -m Qwen3.8-4B-Distill-Decensored-MTP-Q4_K_M.gguf \
  --mmproj Qwen3.8-4B-Distill-Decensored-MTP-mmproj-f16.gguf \
  --image photo.jpg -p "Describe this image."

Do not pass a second -md draft model — the draft head is inside the file.
--spec-type draft-mtp is verify-then-accept, i.e. lossless: the precision of
the MTP head affects only the acceptance rate, never the output distribution.

Sampling recommended by the upstream model: temperature=0.6, top_p=0.95, top_k=20.
Every answer opens with a <think> block — allow generous max_new_tokens.

Measured results

Decensoring (Heretic v2.1.0.dev0, 8 Optuna TPE trials, stock ARA)

Heretic's own in-run scoring — KeywordRate on mlabonne/harmful_behaviors
test[:16] and KLDivergence on mlabonne/harmless_alpaca test[:16]:

Trial layer range Refusals KL divergence
baseline — 16/16 0
1 (selected) 15–22 0/16 0.0407
2 16–27 0/16 0.1412
3 0–18 0/16 14.8663
4 16–28 0/16 0.0506
5 12–20 14/16 0.0026
6 9–23 5/16 0.0293
7 4–17 7/16 0.1599
8 10–27 4/16 0.0470

sorted_index = 0 (trial 1) was selected and is a genuine optimum this round:
0/16 refusals at KL 0.0407. It is not one of the "0 refusals but KL blows up"
trials — trial 3 shows exactly that failure mode (KL 14.87), which is why the
Pareto front was inspected by hand instead of trusting index order.

Re-evaluation at community scale (100 prompts, this build)

Heretic's search uses 16 prompts; that is enough to rank trials but not enough to
publish. So the exported model was re-scored on the larger splits with the same
markers, same normalization, same system prompt:

Model Refusals (harmful_behaviors test[:100]) KL vs base (harmless_alpaca test[:100])
empero-ai/Qwen3.8-4B-Distill (BF16, base) 98/100 0 (by definition)
this decensored build 10/100 0.0407

Greedy decoding, max_new_tokens=64, identical prompts. Both numbers come from
eval_official.py; the raw responses are reproducible with it.

Read the 10/100 honestly. None of them is an empty or broken reply — they are
substantive answers that still open with a ⚠️ Disclaimer / for educational purposes only paragraph, and "disclaimer" is one of Heretic's refusal markers. If
you only count hard refusals ("I can't help with that"), this build produces far
fewer than 10. Marker-based counting is coarse in both directions; the useful
signal is 98 → 10, not the exact denominator.

For reference, the two community decensors of the same base report, at a different
prompt budget and with 2.1.0.dev0-incompatible Heretic versions:
valiolla/Qwen3.8-4B-Distill-heretic 24/100 refusals, KL 0.0109 (Heretic
v1.4.0) and insraq/Qwen3.5-4B-EmperoAI-Qwen3.8-Distill-Heretic-Abliterated
6/100 refusals, KL 0.0167 (v1.4.0). Those ran classic directional ablation;
this build runs ARA, trades a little more drift for fewer refusals, and its KL is
measured on the same 100 prompts as the base rather than on an internal split.

Quantization quality

Perplexity, llama-perplexity -c 512 --chunks 4 on perplexity-corpus.txt
(9,726 B, disjoint from the imatrix corpus):

Model PPL
BF16 GGUF (33 blocks, built-in MTP) 6.4981 ± 0.54774
Q4_K_M, MTP layer Q8_0 (this repo) 6.7652 ± 0.57518
Q4_K_M, MTP layer untouched (control) 6.7652 ± 0.57518

+4.11% relative, which is the honest number for a 4B at ~5.3 BPW. The two Q4
variants are identical to four decimals because perplexity never touches the MTP
head — it is a draft-only layer, so its precision cannot move PPL. That is a
property, not a measurement error, and it is also why MTP precision must be judged
by acceptance rate instead.

Item Value
GGUF Qwen3.8-4B-Distill-Decensored-MTP-Q4_K_M.gguf
Size / sha256 2,869,418,944 B / 446b74134e8d24dbf400900b4fb6344257646e956aae9ece3960de5d46c4ceed
BPW / file_type 5.29 / 15 (MOSTLY_Q4_K_M)
Tensors 393, block_count=33, nextn_predict_layers=1
Type distribution Q4_K:168 / F32:184 / Q6_K:33 / Q8_0:8; blk.32.* = Q8_0:8, F32:7
mmproj (F16) 672,422,944 B, 298 tensors, sha256 4bba1886…
BF16 intermediate 8,665,617,152 B, 393 tensors
imatrix --nextn, 16 chunks × 256 ctx, 6,514,592 B, corpus 614,356 B
ARA diff 14 matrices changed, layers 15–28; patch shard 455.0 MiB; 15 mtp.* untouched
Online check first 64 MiB of each published GGUF byte-identical to local

Draft acceptance A/B (MTP Q8_0 vs MTP Q4_K)

llama-server --spec-type draft-mtp, greedy, 6 prompts × 256 tokens, 8 threads,
acceptance read from the server's slot print_timing:

MTP precision File size Acceptance Mean accepted length Gen speed
Q8_0 (this repo) 2.87 GB 1025/1501 = 68.3% 3.06 15.7 tok/s
Q4_K (default) 2.82 GB 1015/1533 = 66.2% 3.00 15.0 tok/s

Read this table skeptically too: single-run acceptance swings between 57% and
80% on this prompt set (documented in the raw logs), so a 2.1-point gap over 6
prompts is at the edge of noise. Q8_0 is chosen because it costs 53.5 MiB (1.9%)
and a draft head is exactly where 4-bit hurts first; it is not sold here as a
measured win. If you want the smallest file, drop tensor_types.txt and quantize
blk.32.* with everything else.

Honest limitations

  • Q4_K_M is lossy: +4.11% PPL relative to BF16 on this corpus. imatrix and the
    Q8_0 MTP layer reduce the damage; they do not remove it.
  • Heretic v2.1.0.dev0 is a development snapshot. Full defaults (100 trials, 400
    prompts, 100-token generations) were scaled down to fit 8 CPU cores / 16 GiB —
    this is an 8-trial study, not a converged one. A longer run would very likely
    find a better Pareto point.
  • The refusal metric is keyword-based (Heretic's own markers). "Disclaimer" counts
    as a refusal, so the score penalizes cautious-but-compliant answers.
  • No standard benchmark was run. The base model's card reports MMLU-CoT 0.553 and
    GSM8K-CoT 0.785 for the full-precision aligned model; those do not transfer
    to this quant, and this decensor does not claim to preserve them.
  • Decensoring deliberately weakens safety alignment. Compliance risk is yours.
  • The vision projector is the base model's tower, converted to F16; the upstream
    fine-tune is text-only and its vision behavior was not evaluated upstream either.

Credits & license

  1. Qwen team (Alibaba) — Qwen3.5-4B base model family, Apache-2.0.
  2. Empero — empero-ai/Qwen3.8-4B-Distill, full-parameter distillation of
    Qwen3.8 2.4T-A95B into Qwen3.5-4B (~45k teacher traces), Apache-2.0. Direct
    source of all weights, including the 15 MTP tensors and the vision tower.
  3. p-e-w (Philipp Emanuel Weidmann) — Heretic v2.1.0.dev0 (ARA + Optuna TPE).
  4. mlabonne — harmless_alpaca / harmful_behaviors probe datasets.
  5. joeblowma and other community GGUF builders — the practice of keeping the
    MTP layer at higher precision to stabilize draft acceptance.
  6. ggml-org / llama.cpp — GGUF conversion, imatrix, quantization, runtime.
  7. hUh63 — decensor run, MTP-preserving rebuild, quantization and publishing.

License: Apache-2.0, inherited from the base model lineage. Decensoring and
quantization add no new restrictions.

README history 6 versions

The author's README evolved over time. Click a version to see its content at that point.

  1. 2026-11-14 r3: README.md 12c6692 10.4 KB
    Loading...
  2. 2026-11-14 r3: README.md 63e3984 10.4 KB
    Loading...
  3. 2026-11-14 Add README.md d240dd1 9.9 KB
    Loading...
  4. 2026-11-14 Add README.md ad64fee 9.9 KB
    Loading...
  5. 2026-11-13 Upload README.md with huggingface_hub 3102f8c 13.6 KB
    Loading...
  6. 2026-11-13 Upload README.md with huggingface_hub 014253b 13.6 KB
    Loading...
Catalog is the map. Apps are the tools.

Run models on your own machine, not in the cloud.

Every model page has an "Open in Abliteration" button that hands the model directly to the first-party desktop client, at the quantization your rig can actually run. No API keys, no subscription, no prompt leakage.

Open in Abliteration