license: apache-2.0
base_model: huihui-ai/Huihui-Qwen3.8-27B-abliterated
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- vllm
- fp8
- compressed-tensors
- abliterated
- qwen3_5
Huihui-Qwen3.8-27B-abliterated-FP8
FP8 quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated,
produced with llm-compressor (FP8_DYNAMIC).
At the time of quantization huihui-ai published only the BF16 weights and a GGUF
build for this model — no FP8 — so this repo fills that gap for vLLM users.
Scheme
| Weights | FP8 e4m3, per-channel (static) |
| Activations | FP8 e4m3, dynamic per-token |
| Format | compressed-tensors (float-quantized) |
| Calibration | none required (data-free pipeline) |
256 dense Linear modules are quantized. Everything the upstreamQwen/Qwen3.8-27B-FP8 release leaves alone is left in BF16 here too:
linear_attn.*— the hybrid Mamba projections (in_proj_qkv,in_proj_a,in_proj_b,in_proj_z,out_proj)- the whole vision tower (
visual.blocks.*,visual.merger.*) embed_tokens,lm_head, and all norms- the MTP drafter (
mtp.*) — kept in BF16 rather than FP8, so speculative decoding still works
Note the upstream FP8 release uses per-tensor weight scales; this one uses
per-channel, which is a slightly finer-grained (and therefore more accurate)
scheme at the same size.
Serving with vLLM
vllm serve batsclamp/Huihui-Qwen3.8-27B-abliterated-FP8 \
--max-model-len 262144 \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--override-generation-config '{"temperature": 1.0, "top_p": 0.95, "top_k": 20}'
Sampling defaults follow the upstream model card's thinking-mode recommendation
(temp 1.0 / top_p 0.95 / top_k 20).
Caveats
- This is an abliterated model: refusal behaviour has been removed upstream. Safety properties are not those of the original Qwen release.
- Quantization was verified by generation, not by a benchmark suite; no perplexity or eval numbers are claimed.