license: mit
base_model: huihui-ai/Huihui-Qwable-3.6-27b-abliterated-MTP-GGUF
base_model_relation: quantized
tags:
- gguf
- llama.cpp
- qwen3
- mtp
- abliterated
- quantized
Huihui-Qwable-3.6-27b-abliterated Q4_K_M (MTP, GGUF)
A Q4_K_M GGUF quant of huihui-ai/Huihui-Qwable-3.6-27b-abliterated-MTP-GGUF.
Why this quant exists: huihui-ai ships Q4_K_M_Q8 (19.1 GiB) and Q4_K_M_F16 (25.9 GiB). Both are a little too large to leave working room on a 22 GB card once you add the KV cache and compute buffers. This plain Q4_K_M is 15.7 GiB, so it keeps headroom on 22–24 GB GPUs (e.g. RTX 3090 / 4090, or a 22 GB-modded 2080 Ti) while keeping the baked MTP layer intact.
| File | Size | BPW |
|---|---|---|
Huihui-Qwable-3.6-27b-abliterated-Q4_K_M-MTP.gguf |
15.7 GiB | 4.92 |
Lineage
- Base: Mia-AiLab/Qwable-3.6-27b — Qwen3.6-27B with Claude Fable-5 reasoning and assistant traces distilled via full SFT.
- Abliteration + MTP: huihui-ai.
- This quant:
Q4_K_M, produced with llama.cpp (buildb9190) from huihui'sQ4_K_M_F16release.
Speculative decoding (MTP)
The MTP draft head is baked into the model (the nextn layer), so there is no separate draft file. Run it as self-speculation:
llama-server -m Huihui-Qwable-3.6-27b-abliterated-Q4_K_M-MTP.gguf \
-ngl 99 -fa on -c 32768 --parallel 1 --jinja \
--spec-type draft-mtp --spec-draft-n-max 3
Tip: --spec-draft-n-max 3 works better than the larger value suggested upstream. At higher settings the draft head proposes too many tokens, acceptance drops, and throughput ends up lower. Measured on a GB10 (Grace-Blackwell):
--spec-draft-n-max |
decode | draft acceptance |
|---|---|---|
3 |
~18.7 tok/s | ~0.45 |
6 |
slower | ~0.25 |
Acceptance is lower than that of a vanilla Qwen3.6 (~0.75). That gap appears tied to the Fable distill rather than the quantization: changing the draft-head precision (Q4, Q8, or F16) did not materially affect acceptance in testing.
Notes
- Tool calling works with the standard OpenAI-compatible
toolsschema (it returns structuredtool_calls) when thinking is disabled — e.g. pass"chat_template_kwargs": {"enable_thinking": false}in the request. With thinking enabled, the Fable traces can surface internal tool names in the reasoning output. - This is an abliterated model.
- License inherited from the base: MIT.
Credits
Thanks to Mia-AiLab for the Fable distill and huihui-ai for the abliterated MTP build.
📝 Quantized & benchmarked by ai-muninn — writeups on how it was built and how it actually runs.