license: apache-2.0
base_model:

• huihui-ai/Huihui-Qwen3.8-27B-abliterated
  base_model_relation: quantized
  library_name: transformers
  pipeline_tag: text-generation
  tags:
• qwen3_5
• abliterated
• uncensored
• fp8
• w8a8
• compressed-tensors
• mtp
• speculative-decoding
• vllm

Qwen3.8-27B-abliterated-FP8 (MTP preserved)

FP8 W8A8 quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated,
with the multi-token-prediction (MTP) head intact so vLLM's speculative decoding actually works.

31.2 GB, down from 55.6 GB at BF16. Fits one 48 GB card with room for a long-context KV pool,
or leaves ~65 GB of KV headroom on a 96 GB card.

Why this exists

Qwen3.8-27B ships an MTP head (mtp_num_hidden_layers: 1, 15 mtp.* tensors). Under vLLM
speculative decoding it is worth roughly 2.5x tokens per forward pass — on this build,
51.2% draft acceptance and 2.54 tokens committed per round.

Most conversions lose it silently. Qwen3_5ForConditionalGeneration does not instantiate
the MTP module, so from_pretrained() discards all 15 mtp.* tensors as unexpected keys and
save_pretrained() never writes them back. Anything that round-trips the model through
transformers — including several abliterated re-uploads currently on the Hub — comes out with
1184 tensors instead of 1199 and no speculation available, with nothing in the logs to say so.

This build keeps them, and declares them correctly so vLLM can load them (that declaration is a
second, separate trap — see Gotchas).

Running it

vllm serve voska/Qwen3.8-27B-abliterated-FP8-MTP \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 8 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --trust-remote-code

Three of those flags are not optional. vLLM defaults them off, and if you are migrating from
llama.cpp — which handles all three natively — you will ship regressions without noticing.

--reasoning-parser qwen3

Without it the chain-of-thought lands in content along with a stray closing </think>:

"We need to answer user: ... Final: 4.\n</think>\n\n4"

With it, content is "4" and reasoning goes to its own field. This affects every reply.

--enable-auto-tool-choice --tool-call-parser qwen3_xml

Without these, any request with tool_choice: "auto" fails with HTTP 400:

"auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set

Plain chat keeps working perfectly, so health checks and manual testing sail right past it while
every agentic client is hard-broken. qwen3_xml is correct because this model's chat template
emits the XML-nested form, not Hermes JSON. Verify for yourself:

grep -c '<function=' chat_template.jinja   # non-zero -> qwen3_xml

--speculative-config

This is what activates the MTP head. Omit it and you get a perfectly good FP8 model with the
head sitting unused.

For text-only serving add --limit-mm-per-prompt '{"image":0,"video":0}'; the vision tower is
present but left at BF16.

Verify speculation is actually working

Speed alone will not tell you. Check the counter:

curl -s localhost:8000/metrics | grep spec_decode_num_accepted_tokens_total

If it reads 0.0 while num_draft_tokens_total climbs, the head is loaded but every draft is
being rejected — you are paying for speculation and getting nothing back.

Performance

Measured on 1x RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB GDDR7, SM120),
vLLM 0.26.0, speculation at k=3.

Important caveat: this card was running at a 260 W power cap, which pins the SM clock around
952 MHz against a 3090 MHz maximum — roughly 31% of rated clock, with memory near full speed.
Decode is bandwidth-bound so it suffers little; prefill is compute-bound and is being heavily
throttled here. On an unconstrained card, expect prefill materially higher than these numbers.

Aggregate scaling is near-linear — 67 / 117 / 234 / 427 tok/s at 1 / 2 / 4 / 8 concurrent, zero
errors at every level. Per-stream throughput only falls ~17% going from 1 to 8 concurrent users.

MTP: 51.2% draft acceptance, 2.54 tokens/round.

Context is 262,144 native (max_position_embeddings), not YaRN-extended. The architecture is
hybrid — layer_types is full_attention every 4th layer, 16 of 64 — so KV costs only
64 KB/token, which is why a 262K window is affordable at all.

How we actually run it

Served to a small group of concurrent users behind an OpenAI-compatible gateway, on one
dedicated GPU, as a systemd-managed container:

• --max-num-seqs 8 — matched to the real user count rather than left at the default
• --enable-chunked-prefill is doing heavy lifting. This replaced a llama.cpp deployment
  where one user pasting a 130K-token context would monopolize the GPU and starve everyone
  else's decode down to single-digit tok/s. Chunked prefill with decode priority is most of why
  per-stream throughput barely moves as concurrency rises.
• --enable-prefix-caching — large shared system prompts across agent turns
• speculation at k=3, not higher. vLLM warns that num_speculative_tokens > 1 runs multiple
  forward passes over the same single MTP layer, and acceptance decays per position — we measure
  70% / 45% / 28% at positions 0 / 1 / 2. k=3 still earns its keep; deeper does not, and reports
  elsewhere suggest shallower is better again at very long context.
• Restart cap. Whatever supervises this, bound it. A crash-looping vLLM container will
  happily accumulate thousands of restarts overnight while looking "up" from outside.

Migration note: coming from llama.cpp, remember its -c is a total KV budget divided across
-np slots, whereas vLLM's --max-model-len is per request against a shared paged pool. The
pool here (820K tokens) is smaller than a naive slots×context figure but far better utilized.

Quality

FP8 is a lateral move from 8-bit GGUF, not a step down. Published KL-divergence measurements for
this model family put Q8_0 at 0.00064 against BF16, with all 8-bit formats clustered around
0.0006–0.0007 — while the best 4-bit sits at 0.00835, an order of magnitude worse. If you care
about output quality, 8-bit is the floor worth holding.

Abliteration behaviour is unchanged from the base: a 10-prompt refusal probe spanning adult
fiction, profanity, contested political argument, harm reduction, security education, blunt
medical advice and dark fiction returned 0/10 refusals, identical to the BF16 source.

How it was made

FP8 dynamic requires no calibration data — weights are quantized statically per channel,
activations dynamically per token at runtime. That is the whole recipe:

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head", "re:.*visual.*"],
)
oneshot(model=model, recipe=recipe)
model.save_pretrained(DST, save_compressed=True, save_original_format=False)

Roughly 20 seconds on a free GPU. lm_head and the vision tower stay BF16, matching Qwen's own
FP8 release.

Gotchas if you build your own

1. save_pretrained crashes on transformers 5.14.1. You get
ValueError: dictionary update sequence element #0 has length 1; 2 is required, reported
misleadingly as a weight-conversion/offloading failure. The cause is misplaced braces upstream:
weight_map.update({k: ...} for k in ...) builds a generator of one-key dicts instead of a dict
comprehension. Pass save_original_format=False.

2. The MTP head needs grafting and declaring. After quantizing, copy the 15 mtp.*
tensors over from the BF16 source (~0.85 GB, fine to leave unquantized) — and add their 8
Linear modules to quantization_config.ignore:

mtp.fc
mtp.layers.0.mlp.{down,gate,up}_proj
mtp.layers.0.self_attn.{q,k,v,o}_proj

Skip that second step and vLLM expects weight_scale tensors that do not exist. It loads
without complaint and acceptance drops to exactly zero — measured here as 7,194 draft tokens
generated, 0 accepted, with decode falling from ~79 to 30 tok/s. It reads as "disappointing"
rather than "broken," which is what makes it dangerous.

Limitations

• Vision is untested. The tower is present and left at BF16, but this build was only
  exercised for text.
• Long-context behaviour was verified for KV allocation and prefill throughput, not for
  retrieval quality at depth.
• Benchmarks come from one power-limited card; absolute numbers will differ on yours.

About abliteration

This is an uncensored model — refusal behaviour has been ablated. It will engage with material
its instruction-tuned parent declines. Abliteration also removes safety behaviours you may
actually want. Evaluate it for your use case; you own what you deploy.

Credits

• Qwen — base model
• huihui-ai — abliteration
• llm-compressor / vLLM — quantization and serving

Apache 2.0, inherited from the base model.