license: apache-2.0
library_name: transformers
base_model:
- huihui-ai/Huihui-Qwen3.8-27B-abliterated
base_model_relation: quantized
pipeline_tag: image-text-to-text
tags: - fp8
- vllm
- abliterated
- uncensored
- qwen3
Huihui-Qwen3.8-27B-abliterated-FP8
Block-wise FP8 (e4m3) quantization of
huihui-ai/Huihui-Qwen3.8-27B-abliterated,
an abliterated version of Qwen/Qwen3.8-27B.
52 GiB (BF16) → 29 GiB (FP8). The vision tower, all normalization layers,
and the MTP head's own norms/fusion layer stay in BF16, so vLLM's
multimodal path and MTP speculative decoding both keep working.
Quantization details
The layout is identical to the official FP8 release of this model
(Qwen/Qwen3.8-27B-FP8):
| Method | quant_method: fp8, DeepSeek-style block-wise |
| Block size | weight_block_size: [128, 128] |
| Scales | amax / 448.0, stored as bf16 under <module>.weight_scale_inv |
| Activation | dynamic |
| Quantized modules | 407 (MLP gate/up/down_proj, attention q/k/v/o_proj, linear-attn in_proj_qkv / in_proj_z / out_proj, and the MTP head's self_attn/mlp projections) |
Unlike the earlier Qwen3.6 FP8 releases, the official Qwen3.8 FP8 checkpoint
also quantizes part of the MTP head (mtp.layers.0.self_attn.* andmtp.layers.0.mlp.*) — this release matches that. The quantized-module set
is taken verbatim from the reference checkpoint rather than chosen by hand,
so it tracks whatever the official release does.
Deliberately left in BF16 — quantizing these breaks the model:
- Gated DeltaNet internals:
conv1d,in_proj_a,in_proj_b,A_log,dt_bias,norm - All
input_layernorm/post_attention_layernorm/q_norm/k_norm lm_head,embed_tokens- The entire vision tower (
model.visual.*) - The MTP head's own norms and
mtp.fc(fusion layer)
Verification
The quantizer was validated against Qwen/Qwen3.8-27B-FP8 (the FP8 release
of this model's unablated parent). Per the base model's card, abliteration
left the first 15 layers untouched (MTP and the vision tower were not
modified at all). The verification confirms this precisely:
layer 0 / layer 3 (pre-ablation, incl. MTP) — 10/10 probes bit-identical to reference
layer 19 / 39 / 59 (post-ablation, self_attn / mlp) — differ, as expected
Tensor names, dtypes and shapes all match the reference (1199 BF16 + 407 FP8
= 1606 tensors).
Usage (vLLM)
vllm serve mashima/Huihui-Qwen3.8-27B-abliterated-FP8 \
--quantization fp8 \
--dtype bfloat16 \
--tensor-parallel-size 2 \
--max-model-len 262144 \
--kv-cache-dtype fp8 \
--trust-remote-code \
--reasoning-parser qwen3 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
Tested on vLLM v0.24.0.
Chat template / thinking
Uses the model's bundled chat_template.jinja, which is reasoning_effort-based
(low / medium / high / xhigh) rather than the older enable_thinking
boolean. Pass e.g. "chat_template_kwargs": {"reasoning_effort": "low"} in
requests, or set --default-chat-template-kwargs server-side.
⚠️ Known issue observed on this checkpoint: with --reasoning-parser qwen3,reasoning_content comes back empty in both streaming and non-streaming
responses, even though the model is actually thinking (verified via the raw/v1/completions endpoint, which shows a normal <reasoning>...</think>
generation). Feeding the exact same generated text to the parser directly
(outside the server) extracts reasoning/content correctly, so the bug looks
like it's in how vLLM v0.24.0's decode loop drives the parser during
generation, not in the parser's extraction logic itself. Not root-caused
further — only tested on this FP8 checkpoint, so whether it also reproduces
on the unquantized model is unconfirmed. Answer quality is unaffected either
way; you just don't get the reasoning trace back over the API.
Hardware notes
Verified on 2× RTX 3090 (Ampere, sm_86) with TP=2, ~22 GB per card at--gpu-memory-utilization 0.92 and 262K context with fp8 KV cache.
Ampere has no FP8 tensor cores, so vLLM falls back to the Marlin W8A16
kernel (weight-only decompression). You get the memory saving but no FP8
compute speedup — expect this warning, which is normal:
Your GPU does not have native support for FP8 computation but FP8 quantization
is being used. Weight-only FP8 compression will be used leveraging the Marlin kernel.
On Ada / Hopper / Blackwell (sm_89+) FP8 runs natively.
MTP speculative decoding works: measured mean acceptance length 2.22 on 2× 3090.
Inherited warnings from the base model
This is a quantization of an abliterated (uncensored) model. The safety
filtering of the base model has been significantly reduced. The warnings from
huihui-ai
apply unchanged:
- Risk of sensitive or controversial outputs — review generated content carefully
- Not suitable for all audiences — may be inappropriate for public settings or underage users
- Legal and ethical responsibility rests with the user — ensure compliance with local law
- Intended for research, testing and controlled environments, not unsupervised production use
- No default safety guarantees — this model has not undergone safety optimization
Quantization does not change the behaviour of the base model in this respect.
Credit for the base model and its abliteration goes to
huihui-ai; please support their work there.