license: apache-2.0
library_name: transformers
base_model:
- huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated
base_model_relation: quantized
pipeline_tag: image-text-to-text
tags: - fp8
- vllm
- abliterated
- uncensored
- qwen3_6
- thinkingcap
- token-efficient
Huihui-ThinkingCap-Qwen3.6-27B-abliterated-FP8
Block-wise FP8 (e4m3) quantization of
huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated,
which is in turn an abliterated version of
bottlecapai/ThinkingCap-Qwen3.6-27B.
52 GiB (BF16) → 29 GiB (FP8), with the vision tower, the MTP head and all
normalization layers kept in BF16 so that vLLM's multimodal path and MTP
speculative decoding keep working.
Quantization details
The layout is identical to the official FP8 releases of this architecture
(Qwen/Qwen3.6-27B-FP8, bottlecapai/ThinkingCap-Qwen3.6-27B-FP8):
| Method | quant_method: fp8, DeepSeek-style block-wise |
| Block size | weight_block_size: [128, 128] |
| Scales | amax / 448.0, stored as bf16 under <module>.weight_scale_inv |
| Activation | dynamic |
| Quantized modules | 400 (MLP gate/up/down_proj, attention q/k/v/o_proj, linear-attn in_proj_qkv / in_proj_z / out_proj) |
Deliberately left in BF16 — quantizing these breaks the model:
- Gated DeltaNet internals:
conv1d,in_proj_a,in_proj_b,A_log,dt_bias,norm - All
input_layernorm/post_attention_layernorm/q_norm/k_norm lm_head,embed_tokens- The entire vision tower (
model.visual.*) - The MTP head (
mtp.*) — kept unquantized so vLLM's--speculative-configworks
The set of quantized modules was taken verbatim from the reference FP8
checkpoint rather than chosen by hand.
Verification
The quantizer was validated against bottlecapai/ThinkingCap-Qwen3.6-27B-FP8
(the FP8 release of this model's unablated parent). For modules that
abliteration does not touch, the output is bit-for-bit identical to the
reference checkpoint:
layers.3.mlp.gate_proj bit-identical: weight=True scale=True
layers.3.mlp.up_proj bit-identical: weight=True scale=True
layers.3.self_attn.q_proj bit-identical: weight=True scale=True
layers.0.linear_attn.in_proj_qkv bit-identical: weight=True scale=True
layers.0.linear_attn.in_proj_z bit-identical: weight=True scale=True
layers.3.mlp.down_proj differs (abliteration target)
layers.3.self_attn.o_proj differs (abliteration target)
layers.0.linear_attn.out_proj differs (abliteration target)
Only the projections abliteration actually modifies differ — exactly as expected.
Tensor names, dtypes and shapes all match the reference (1199 BF16 + 400 FP8 = 1599 tensors).
Usage (vLLM)
vllm serve mashima/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-FP8 \
--quantization fp8 \
--dtype bfloat16 \
--tensor-parallel-size 2 \
--max-model-len 262144 \
--kv-cache-dtype fp8 \
--trust-remote-code \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
Tested on vLLM v0.24.0.
Hardware notes
Verified on 2× RTX 3090 (Ampere, sm_86) with TP=2, ~22 GB per card at--gpu-memory-utilization 0.92, 262K context with fp8 KV cache.
Ampere has no FP8 tensor cores, so vLLM falls back to the Marlin W8A16 kernel
(weight-only decompression). You get the memory saving but no FP8 compute
speedup — expect this warning, which is normal:
Your GPU does not have native support for FP8 computation but FP8 quantization
is being used. Weight-only FP8 compression will be used leveraging the Marlin kernel.
On Ada / Hopper / Blackwell (sm_89+) FP8 runs natively.
MTP speculative decoding works: measured mean acceptance length 2.42 on 2× 3090.
Inherited warnings from the base model
This is a quantization of an abliterated (uncensored) model. The safety
filtering of the base model has been significantly reduced. The warnings from
huihui-ai
apply unchanged:
- Risk of sensitive or controversial outputs — review generated content carefully
- Not suitable for all audiences — may be inappropriate for public settings or underage users
- Legal and ethical responsibility rests with the user — ensure compliance with local law
- Intended for research, testing and controlled environments, not unsupervised production use
- No default safety guarantees — this model has not undergone safety optimization
Quantization does not change the behaviour of the base model in this respect.
Credit for the base model and its abliteration goes to
huihui-ai; please support their work there.