base_model: huihui-ai/Huihui-Qwen3.8-27B-abliterated
base_model_relation: quantized
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- fp8
- fp8-dynamic
- llm-compressor
- compressed-tensors
- vllm
- mtp
- speculative-decoding
Huihui-Qwen3.8-27B-abliterated-FP8-dynamic
FP8 (W8A8, dynamic per-token activations / per-channel weights) quantization of
huihui-ai/Huihui-Qwen3.8-27B-abliterated,
produced with llmcompressor 0.13.0, with the MTP (multi-token prediction) head preserved so
vLLM speculative decoding (--speculative-config '{"method":"mtp","num_speculative_tokens":1}') works.
What is quantized
Only the 256 Linear modules of the language model's full-attention and MLP paths are FP8.
Left in BF16: the vision tower (model.visual.*), the GatedDeltaNet linear-attention layers
(*.linear_attn.*), lm_head, embeddings, and the whole mtp.* head.
Recipe (data-free, no calibration set)
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
MODEL_ID = "huihui-ai/Huihui-Qwen3.8-27B-abliterated"
model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID, dtype="bfloat16")
processor = AutoProcessor.from_pretrained(MODEL_ID)
recipe = QuantizationModifier(
targets="Linear", scheme="FP8_DYNAMIC",
ignore=["re:.*lm_head", "re:.*embed_tokens$", "re:.*visual.*", "re:.*model.visual.*",
"re:.*linear_attn.*", "re:^mtp.*", "re:.*mtp.*"],
)
oneshot(model=model, recipe=recipe)
model.save_pretrained("out-fp8"); processor.save_pretrained("out-fp8")
MTP graft
Qwen3_5ForConditionalGeneration does not instantiate the mtp.* submodule, so save_pretrained
silently drops the 15 MTP tensors. After quantization they were copied verbatim (BF16) out of the
source shards into a dedicated shard, model.safetensors.index.json was regenerated to cover them, andre:.*mtp.* was added to quantization_config.ignore. A copied-in shard without an index entry is
ignored at load time — the drafter then runs on random weights and acceptance drops to ~0%.
Verification
vLLM 0.28.0, single RTX PRO 6000 Blackwell 96GB:
vllm serve <model> --max-model-len 32768 --max-num-seqs 256 \
--speculative-config '{"method":"mtp","num_speculative_tokens":1}'
Resolved architecture: Qwen3_5MTP,Detected MTP model. Sharing target model embedding weights with the draft model.- Greedy 300-token generation:
spec_decode_num_accepted_tokens_total141 /num_draft_tokens_total181 = 77.9% acceptance, confirming the grafted MTP weights are real.
Note: on a 96GB card the default max_num_seqs=1024 exceeds the available Mamba cache blocks and boot
fails; pass --max-num-seqs 256 (or lower) for the hybrid GatedDeltaNet layers.