library_name: gguf
license: apache-2.0
license_link: https://huggingface.co/iBotIA/Qwen3.8-4B-Empero-AI-FullStack/raw/main/README.md
base_model:
- iBotIA/Qwen3.8-4B-Empero-AI-FullStack
- empero-ai/Qwen3.8-4B-Distill
- Qwen/Qwen3.5-4B
base_model_relation: quantized
quantized_by: hUh63
language: - en
- zh
- ko
- ja
- ru
- es
- fr
pipeline_tag: image-text-to-text
tags: - gguf
- llama.cpp
- quantization
- q4_k_m
- imatrix
- mtp
- multi-token-prediction
- speculative-decoding
- qwen3_5
- qwen3.8
- image-text-to-text
- vision
- full-stack
- typescript
- react
- nestjs
- flutter
- decensored
- uncensored
- abliterated
- heretic
- heretic-2.1.0.dev0
- ara
Qwen3.8-4B-Empero-FullStack — Decensored · built-in MTP · Q4_K_M GGUF
GGUF quantizations of iBotIA/Qwen3.8-4B-Empero-AI-FullStack,
decensored with Heretic v2.1.0.dev0 and converted with llama.cpp.
The upstream model is a full-stack engineering fine-tune (279k curated segments: React Router v8,
NestJS, Prisma / Drizzle, TypeScript strict mode, Tailwind + shadcn, Flutter) of
empero-ai/Qwen3.8-4B-Distill — a
full-parameter distillation of the Qwen3.8 2.4T-A95B teacher into the
Qwen3.5-4B architecture. Text weights and the MTP
head come from that lineage; the vision tower is the Qwen3.5 tower carried along with it.
[!WARNING]
This build deliberately weakens the source model's safety alignment. Research and local
use only — compliance risk is yours. Q4_K_M is lossy. No standard benchmark suite (MMLU,
GSM8K, HumanEval, …) was run on this derivative.2.1.0.dev0is a Heretic development
snapshot, not a stable release.
Files
| File | Quant | Size | Notes |
|---|---|---|---|
Qwen3.8-4B-Empero-FullStack-Decensored-MTP-Q4_K_M.gguf |
Q4_K_M, MTP layer Q8_0 | 2,869,418,944 B | recommended; ~5.29 BPW |
Qwen3.8-4B-Empero-FullStack-Decensored-MTP-mmproj-f16.gguf |
F16 vision projector | 672,422,944 B | required for image input |
Two comparison builds were produced in the same run and are documented here rather than
shipped, to keep the repo small: Q4_K_M with the MTP layer at Q6_K (2,840,214,464 B B,
5.23 BPW) and plain Q4_K_M with no per-tensor override (2,815,884,224 B B).
Their acceptance-rate measurements are in spec_*.json.
SHA256:
ebd5efb2221b07d702cdb7ad351e25cb8ae9810ab2b25773bbdcc21719272b4e Qwen3.8-4B-Empero-FullStack-Decensored-MTP-Q4_K_M.gguf
4bba18869b9f61cdbd31d0bbe047994c343a82af8ac2e5371f3d98f5ead8c220 Qwen3.8-4B-Empero-FullStack-Decensored-MTP-mmproj-f16.gguf
What "MTP layer at high precision" means here
The source ships 15 mtp.* tensors (mtp_num_hidden_layers = 1) — a multi-token-prediction
draft head inside the same network. convert_hf_to_gguf.py maps them onto block 32, so the
GGUF has block_count = 33 and nextn_predict_layers = 1.
Because speculative decoding is verify-then-accept, the draft head can never change the
output distribution — it only decides how many tokens per verification step get accepted.
Quantizing it therefore shows up as an acceptance-rate effect, not a perplexity effect.
That is exactly why it gets a precision override here:
| Item | Value |
|---|---|
| GGUF | Qwen3.8-4B-Empero-FullStack-Decensored-MTP-Q4_K_M.gguf |
| Tensors / blocks | 393 / 33 (nextn_predict_layers = 1) |
| Type distribution | F32:184 / Q4_K:168 / Q6_K:33 / Q8_0:8 |
blk.32.* (the MTP layer) |
Q8_0:8 / F32:7 |
| BPW | 5.29 |
Measured acceptance rate, 12 prompts × 300 tokens each, llama-server --spec-type draft-mtp,
identical sampling and identical trunk weights across the three columns:
| MTP 层精度(其余同为 Q4_K_M) | 池化接受率 | 各 prompt 平均接受率 | 平均接受长度 |
|---|---|---|---|
| Q8_0(本仓库) | 68.75% | 69.2%(57.8–76.75%) | 3.07 |
| Q6_K(备选档) | 67.97% | 68.6%(56.16–76.75%) | 3.05 |
| 默认档(未覆盖,Q4_K/Q6_K 混合) | 67.63% | 68.4%(55.06–77.98%) | 3.04 |
Read the spread, not the ranking: single-run acceptance rates on this machine move by tens of
percentage points between runs, so a one-digit difference between columns is noise. The
numbers above are reported with their min/max so you can judge that yourself.
Quantization quality
Perplexity with llama-perplexity -c 512 --chunks 4 on a held-out corpus written for this
run (disjoint from the imatrix calibration text):
| Model | PPL | Δ vs BF16 |
|---|---|---|
| BF16 GGUF (33 blocks, built-in MTP) | 10.0916 ± 1.14470 | — |
| Q4_K_M, MTP Q8_0 (this repo) | 10.3287 ± 1.17320 | +2.35% |
| Q4_K_M, MTP Q6_K | 10.3287 ± 1.17320 | +2.35% |
| Q4_K_M, no MTP override (control) | 10.3287 ± 1.17320 | +2.35% |
The three Q4 variants land within a hair of each other because perplexity never executes the
draft head — it is a draft-only layer. Do not read the PPL table as evidence about MTP
precision; read the acceptance table above.
Decensoring — what was actually run
Heretic v2.1.0.dev0 (p-e-w/heretic @ 75a640b7eb, stock ARA modifier + stockKeywordRate / KLDivergence scorers, framework source unmodified), driven bysearch_lam.py from this repo. Machine: 24 cores / 48 GiB CPU, no GPU.
Protocol — this is the published one, applied identically to the base model and to every
candidate:
- refusals:
mlabonne/harmful_behaviorstest[:100], greedy, 64 new tokens - drift: KL divergence of first-token distributions on
mlabonne/harmless_alpacatest[:100]
against the unmodified BF16 source - refusal detection: Heretic's own keyword markers, so a compliant answer that opens with
⚠️ Disclaimercounts as a refusal. The drop from the base rate is the signal, not the exact
denominator.
Method: ARA (Arbitrary-Rank Ablation) direction + scalar magnitude λ. ARA produces the
ablation direction as LoRA adapters with lora_alpha = lora_rank = 16, so the deployed scaling
is alpha/r = 1 and lora_B is exactly the task vector's magnitude. For each of
3 directions this run fitted the direction once, then sweptW_eff(λ) = W_base + λ·(B@A), bounded λ by bisection at the KL budget, and scored only on the
100-prompt protocol. A deterministic smoke test (KL(λ=0) ≈ 0) guards against a mis-scaled
adapter.
| 方向 | λ 曲线(λ → KL) |
|---|---|
A_preserve082 |
1.00 → 0.0145, 1.40 → 0.0246, 1.80 → 0.0364, 2.20 → 0.0494, 2.60 → 0.0677 |
Results
| Model | Refusals | KL vs base |
|---|---|---|
iBotIA/Qwen3.8-4B-Empero-AI-FullStack (BF16 source) |
97/100 | 0 (by definition) |
this build (direction A_preserve082, λ = 1.6362) |
14/100 | 0.0308 |
| target | ≤ 15/100 | ≤ 0.04 |
Verdict: 目标达成:14/100 拒答、KL 0.0308,两个约束同时满足。
| # | 方向 | λ | 拒答 | KL | 判定 |
|---|---|---|---|---|---|
| 200 | A_preserve082 |
1.6362 | 14/100 | 0.0308 | HIT |
| 201 | A_preserve082 |
1.925 | 14/100 | 0.0399 | HIT |
共 2 个 100 条全量真评点。KL(λ) 曲线见仓库里的 curve-*.json(每个方向一份),全部原始点见 points.jsonl,没有丢弃任何点。
Pareto front (non-dominated, refusals × KL):
拒答 14/100 KL=0.0308 (点 200, 方向 A_preserve082, λ=1.6362)
Prompt format
The chat template is inherited verbatim from the source. System message first, add_generation_prompt
opens a <think> block, so every answer starts with reasoning — give it room
(max_new_tokens ≥ 512 for anything non-trivial).
Upstream-recommended sampling for this distillation family: temperature=0.6, top_p=0.95,top_k=20. Greedy decoding (temperature 0) is explicitly discouraged by the author of the
upstream model — reasoning models in this class fall into repetition loops.
Usage
llama.cpp (built after the MTP-supporting commit; --spec-type draft-mtp is required):
llama-cli -m Qwen3.8-4B-Empero-FullStack-Decensored-MTP-Q4_K_M.gguf --spec-type draft-mtp \
-p "Write a NestJS guard that rejects requests without a valid JWT." \
-n 512 -c 8192 -t 8
Do not pass a second model with -md — the draft head lives inside this file. Without--spec-type draft-mtp the head is loaded but never executed, which is slower than not using
speculation at all.
Vision:
llama-mtmd-cli -m Qwen3.8-4B-Empero-FullStack-Decensored-MTP-Q4_K_M.gguf --mmproj Qwen3.8-4B-Empero-FullStack-Decensored-MTP-mmproj-f16.gguf \
--image screenshot.png -p "Turn this UI mockup into a Tailwind + shadcn component."
llama-server (acceptance rate is only printed by the server's per-slot timing output):
llama-server -m Qwen3.8-4B-Empero-FullStack-Decensored-MTP-Q4_K_M.gguf --spec-type draft-mtp -c 8192 -np 1 --host 127.0.0.1 --port 8080
How this file was produced
Full runbook, scripts and every raw measurement are in the CNB repoOpenOmniBot-omniinfer/Test underdocs/quantization/qwen38-4b-empero-fullstack/ — the method is public, the weights are not
mirrored there.
- Download the 6 BF16 shards (9,319,820,368 B, 738 tensors, 15
mtp.*),
each verified against its Hugging Face LFS SHA256. - Fit ARA directions, sweep λ, score 100 prompts per candidate (2 full points, 2 total records, 0.7 h).
- Rebuild the HF conversion input from the original index: only the 14 matrices
ARA actually changed are written to a patch shard that sorts last; all 15mtp.*tensors, the
whole vision tower, embeddings and every norm stay byte-identical to the source checkpoint
(heretic_ara_report.jsoncarries the per-tensor SHA256 of the MTP head, patch shard
505.0 MiB). convert_hf_to_gguf.py --outtype bf16→ BF16 GGUF with MTP as block 32;--mmproj --outtype f16
for the projector.llama-imatrix --nextnover a ~500 KiB calibration corpus weighted toward this model's actual
domain (60% TypeScript/TSX + Flutter + SQL, 20% multilingual prose, 20% math/reasoning).--nextnis required for the MTP layer to receive importance statistics at all.- Three
llama-quantizepasses (Q8_0 / Q6_K / no override onblk.32.*), header assertions per
build, PPL on a disjoint corpus, acceptance-rate A/B.
Honest limitations
- The refusal target is a keyword count, not a judgment. 目标达成。
- Q4_K_M is lossy: +2.35% perplexity relative to BF16 on this corpus.
- Acceptance-rate A/B is 12 prompts per column. That is enough to see the order of magnitude,
not enough to resolve a 2-point difference — and per-run variance here is large. 2.1.0.dev0is a development snapshot of Heretic; upstream defaults are 100 trials with
400-prompt probes. This run fits a single CPU job.- The source model's own card reports a final training loss of 2.508 on 250 steps over 279k
segments; no independent evaluation of the full-stack ability was performed by me, before or
after decensoring. The only thing measured here is refusals, drift, perplexity and acceptance
rate. - Vision behaviour was not evaluated upstream either; the projector is the base tower converted
to F16, unchanged.
Credits
- Qwen team (Alibaba) — Qwen3.5-4B architecture and vision tower, Apache-2.0.
- Empero-AI —
empero-ai/Qwen3.8-4B-Distill, the distillation all weights derive from. - iBotIA / iWebRoot —
iBotIA/Qwen3.8-4B-Empero-AI-FullStack,
the full-stack fine-tune this repo quantizes. Original model card, training logs and license
live there. - p-e-w (Philipp Emanuel Weidmann) — Heretic v2.1.0.dev0
(ARA + Optuna TPE), AGPL-3.0-or-later. - mlabonne —
harmless_alpaca/harmful_behaviorsprobe datasets. - ggml-org / llama.cpp — GGUF conversion, imatrix, quantization, runtime.
- hUh63 — λ sweep, MTP-preserving rebuild, domain-matched calibration corpus, quantization,
publishing.
License: Apache-2.0, inherited from the base model lineage. Decensoring and quantization add
no new restrictions. Heretic itself is AGPL-3.0-or-later; it was used as a tool, and no Heretic
source is redistributed in the weights.