license: apache-2.0
base_model: huihui-ai/Huihui-Qwen3.8-27B-abliterated
base_model_relation: quantized
pipeline_tag: text-generation
tags:

• awq
• w4a16
• llm-compressor
• compressed-tensors
• vllm
• mtp
• speculative-decoding

Huihui-Qwen3.8-27B-abliterated — AWQ W4A16 (MTP head preserved)

4-bit weight-only AWQ quantization of
huihui-ai/Huihui-Qwen3.8-27B-abliterated,
produced with llm-compressor 0.13.0.
The Multi-Token Prediction (MTP) head is kept in bf16 and shipped with the checkpoint so
vLLM speculative decoding works out of the box.

• Scheme: W4A16_ASYM, group size 128, compressed-tensors pack-quantized format
• Quantized: text-decoder Linear modules in the full-attention layers and all MLPs
• Left in bf16: lm_head, the vision tower (visual.*), the MTP head (mtp.*),
  and every GatedDeltaNet linear-attention projection (linear_attn.*)
• Size: ~27 GB (from ~56 GB bf16)
• Calibration: 128 samples of HuggingFaceH4/ultrachat_200k, max_seq_length=1024

Why parts are left unquantized

The model is a hybrid: of its 64 text layers, 48 are GatedDeltaNet (linear_attn) and 16
are full attention (full_attention_interval=4). AWQ activation smoothing has no
well-formed mapping for the GatedDeltaNet blocks, so those projections are excluded from
both smoothing and quantization rather than approximated. v_proj -> o_proj smoothing is
also skipped on the full-attention layers, whose GQA + output-gate shapes do not line up.

Usage (vLLM)

vllm serve zeallat/Huihui-Qwen3.8-27B-abliterated-AWQ-MTP \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

The log should report a detected MTP model. Dropping --speculative-config runs the model
without speculative decoding.

Note on the MTP tensors

save_pretrained does not carry the MTP head through quantization, so the 15 mtp.*
tensors were copied back from the source checkpoint in bf16, the
model.safetensors.index.json weight map was regenerated to cover them, and
re:.*mtp.* is present in quantization_config.ignore. Without all three, vLLM either
tries to build a quantized drafter and fails, or silently loads an untrained draft head
(0% acceptance).

If you previously served this model and then changed quantization_config, clear the
torch.compile cache before rebooting.

Disclaimer

This is an abliterated (refusal-behaviour-removed) base model. Safety alignment has been
weakened upstream; you are responsible for how you deploy it.