license: apache-2.0
base_model: Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16
base_model_relation: quantized
library_name: vllm
tags:
- qwen3_5
- qwen3.8
- compressed-tensors
- w8a16
- int8
- abliterated
- gfx90a
- mi210
- rocm
Qwen3.8-27B abliterated — W8A16
INT8 W8A16 quantization of Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16, which is
itself an abliterated derivative of Qwen/Qwen3.8-27B.
Produced and verified on 2x AMD MI210 (gfx90a / CDNA2).
On this hardware int8 is SLOWER than BF16 — roughly half the decode rate —
and MTP speculative decoding does not work. Both are measured below. Use these
only if you need the smaller memory footprint.
Scheme
Weights 8-bit, group size 128, symmetric. Activations are not quantized (pack-quantized), so this is weight-only int8.
Quantized with llm-compressor 0.12.1a20260701, QuantizationModifier
(round-to-nearest). No calibration data was used and none is needed: the scheme is weight-only, so scales come straight from the weights.
Measured performance — read this before choosing int8
Benchmarked on 2x AMD MI210 (gfx90a), vLLM 0.27.2rc0+mi210.1, TP=2,
prefix caching disabled (leaving it on inflates decode, because the
harness derives decode time by subtracting a prefill that the second, cached
request never performs).
Decode, tokens/s, at three context depths:
| config | @0 | @8k | @32k | weights |
|---|---|---|---|---|
| BF16 + MTP | 54.1 | 33.3 | 16.0 | 52 GB |
| BF16 | 31.3 | 17.5 | 7.7 | 52 GB |
| W8A8 (this family) | 17.3 | 12.0 | 6.4 | 34 GB |
| W8A16 (this family) | 16.1 | 11.3 | 6.2 | 34 GB |
INT8 is roughly half the decode rate of BF16 on this hardware, not faster.
gfx90a has no fused dequantization path and its INT8 peak equals its BF16 peak
(181 TOPS vs 181 TFLOP/s), so every quantized weight is converted to bf16 before
the MFMA units can use it, and there is no compute headroom to recover that cost
from. The same pattern held for int8 activations and for an int8 KV cache, which
lost 38-48% of throughput at depth.
What int8 does buy is memory: 34 GB against 52 GB, and roughly 830k KV
tokens against 578k at 64K context (12.7x vs 8.8x concurrency). If you are
capacity-bound rather than latency-bound, that is the trade on offer.
Of the two, W8A8 is the faster — about 6% on decode and 36% on prefill —
despite quantizing more. Prefer it over W8A16 unless you specifically need
weight-only.
On a CUDA GPU with real INT8 tensor-core paths the picture is likely different;
none of the above should be read as a claim about other hardware.
MTP does not work on these quants
The base model ships an MTP draft head and mtp_num_hidden_layers: 1, and vLLM
registers qwen3_5_mtp as a speculative method. On BF16 it is a large win —
86-100% draft acceptance, ~2x decode.
On any compressed-tensors quantization of this model it produces 0.0%
acceptance: the draft runs every step and every token is rejected, so it is
pure overhead and leaves you slower than not using it.
Isolated by elimination, all on the same hardware and vLLM build:
| target | acceptance |
|---|---|
| official BF16 | 86-100% |
| abliterated BF16 | 83.8% |
W8A8, mtp.* kept BF16 |
0.0% |
W8A8, mtp.* also quantized |
0.0% |
| W8A16 (weight-only) | 0.0% |
| W8A8 target, draft pointed at a BF16 checkpoint | 0.0% |
So it is not abliteration, not the quantization scheme, not how the MTP head
itself is treated, and not the draft checkpoint path — vLLM reads mtp.* from
the target regardless. Exactly 0.0% rather than a degraded rate points to
something structurally broken rather than quality loss.
Possibly relevant: vLLM copies the target's quantization into the draft config
for the dspark method and has no equivalent for MTP.
Do not pass --speculative-config with these checkpoints.
What is quantized — and what deliberately is not
Qwen3.8-27B is a hybrid: layer_types is 48 linear_attention (GDN) layers
and 16 full_attention, full_attention_interval: 4. Only 16 of 64 layers hold
a KV cache, which is why its context is cheaper than the layer count suggests.
Of 985 modules, 256 are quantized and 729 are left at BF16:
| quantized | count |
|---|---|
model.language_model.layers.N.mlp.{gate,up,down}_proj |
192 (all 64 layers) |
model.language_model.layers.N.self_attn.{q,k,v,o}_proj |
64 (the 16 full-attention layers) |
| protected | count | why |
|---|---|---|
re:.*linear_attn.* |
384 | all 48 GDN layers, incl. conv1d / in_proj_* / out_proj |
re:.*visual.* |
167 | vision tower |
re:.*norm.* |
271 | norms |
re:^mtp\..* |
15 | MTP draft head |
lm_head, embed_tokens |
2 |
MLPs dominate parameter count, so quantizing them captures most of the footprint
reduction while the sensitive linear-attention path stays untouched.
Loading gotcha
On Qwen3_5Config, AutoModelForCausalLM maps to Qwen3_5ForCausalLM — the
text-only class. It loads without error and silently drops all 54 vision
blocks. Use AutoModelForImageTextToText, which maps toQwen3_5ForConditionalGeneration.
Serving (vLLM)
vllm serve davetha/Qwen3.8-27B-abliterated-W8A16 \
--tensor-parallel-size 2 --max-model-len 131072 \
--gpu-memory-utilization 0.85 --trust-remote-code \
--enable-auto-tool-choice --tool-call-parser qwen3_xml \
--reasoning-parser qwen3
qwen3_xml, not hermes: the chat template emits<tool_call><function=name><parameter=x> XML, which the hermes parser does not
match.
On gfx90a note that head_dim = 256, which upstream vLLM's ROCm paged-attention
gate rejects (it accepts only 64 or 128).
Verified
- Loads under vLLM
0.27.2rc0+mi210.1, TP=2 on 2x MI210 - Correct at temperature 0:
17*23->391; 2:15pm-6:40pm ->265minutes;
string-reverse one-liner ->s = s[::-1]
Not measured: perplexity, and no benchmark suite has been run. Quality claims
beyond the spot checks above are unverified.
Note
The base model is abliterated, so the usual refusal behaviour is substantially
removed. You are responsible for how you use it.