base_model:
- lflare/Qwen3.6-27B-uncensored-heretic-v2-mtp
base_model_relation: quantized
license: apache-2.0
pipeline_tag: image-text-to-text
tags: - qwen3_5
- fp8
- compressed-tensors
- mtp
- speculative-decoding
- multimodal
Qwen3.6-27B-uncensored-heretic-v2-mtp-FP8
FP8 (compressed-tensors, FP8_DYNAMIC W8A8) quantisation of lflare/Qwen3.6-27B-uncensored-heretic-v2-mtp,
attnbf16 variant: the entire self-attention path is kept in bf16 and only the MLPs are FP8. Ships a
working MTP self-speculative-decoding head.
What's kept in bf16
lm_head, the MTP head, the vision tower, the whole linear_attn (Gated-DeltaNet / SSM) block,
and the entire self_attn path (incl. the attention output gate fused into q_proj on Qwen3.5/3.6).
Only the ~17 B MLP params are FP8. This keeps quantisation off the multiplicative attention gate and the
16 long-range full-attention layers, at a cost of ~+1.5 GiB (~4.6 %) vs a fully-quantised FP8 build. Seerecipe.yaml.
Accuracy vs the bf16 parent: KLD ≈ 0.0203 nats (kld_heretic_attnbf16.json), measured per-token onneuralmagic/calibration (8 samples, seq 1024). This is markedly lower than a plain-attention FP8 build
of the same base.
Speculative decoding (MTP)
The bf16 MTP head is declared in quantization_config.ignore so vLLM loads it correctly. (A bf16 MTP head
regrafted into a compressed-tensors quant is otherwise mis-loaded and yields 0 % draft acceptance — this
build fixes that.)
Measured MTP acceptance: 73.0 % (vLLM 0.26.0, Blackwell, greedy,--speculative-config '{"method":"mtp","num_speculative_tokens":3}').
vllm serve lflare/Qwen3.6-27B-uncensored-heretic-v2-mtp-FP8 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--max-num-seqs 32
--max-num-seqs 32 (or lower) is required — Qwen3.6 is a hybrid linear-attention model whose Mamba cache
otherwise runs out of blocks at the default max_num_seqs.