license: apache-2.0
base_model: Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16
base_model_relation: quantized
library_name: vllm
tags:
- qwen3_5
- qwen3.8
- compressed-tensors
- w8a8
- int8
- abliterated
- gfx90a
- mi210
- rocm
Qwen3.8-27B abliterated — W8A8 + INT8 GDN projections
INT8 W8A8 quantization of Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16,
itself an abliterated derivative of Qwen/Qwen3.8-27B. Abliterated models have had
refusal behaviour deliberately removed; that property is inherited here unchanged.
Produced and measured on a single AMD MI210 (gfx90a / CDNA2).
What is different from a normal W8A8 of this model
This checkpoint additionally quantizes the three large gated-delta-net (GDN)
projections — in_proj_qkv, in_proj_z, out_proj — across all 48 linear_attention
layers. That is a deliberate departure from the usual recipe for this architecture,
which protects the whole linear-attention path as quantization-sensitive. Read the
evaluation section before using it; a plain W8A8 build is the conservative choice.
It buys, on one MI210:
| W8A8 (GDN bf16) | this checkpoint | |
|---|---|---|
| decode, MTP N=1 | 40.5 tok/s | 50.0 tok/s (+23.5%) |
| per forward step | 43.68 ms | 35.68 ms |
| KV cache capacity | 298,381 tok | 374,711 tok (+26%) |
| on disk | 35 GB | 30 GB |
The GDN projections were 5.54 B parameters sitting in BF16 — 11.07 GB read every decode
step, against 19.3 GB for everything already quantized. On a bandwidth-bound decode that
was ~30% of kernel time in one unquantized tensor family.
What is quantized
Everything the parent W8A8 recipe quantizes (MLPs on all 64 layers, self_attn
projections on the 16 full_attention layers), plus:
| added here | count | |
|---|---|---|
linear_attn.in_proj_qkv |
48 | 10240x5120 |
linear_attn.in_proj_z |
48 | 6144x5120 |
linear_attn.out_proj |
48 | 5120x6144 |
Still BF16, deliberately:
| protected | why |
|---|---|
linear_attn.in_proj_a, linear_attn.in_proj_b |
48x5120 each — 0.42% of the block's bytes, but they feed the recurrent gating (a → softplus → exp → decay, b → sigmoid → beta). Error there compounds along the sequence for no meaningful saving. |
linear_attn.conv1d, all norm |
small, sensitive |
re:.*visual.* |
vision tower |
mtp.* |
MTP draft head, kept BF16 for speculative decoding |
lm_head, embed_tokens |
Scheme
Symmetric per-output-channel INT8 weights, matching the parent checkpoint exactly:
scale = max|W| / 127 # computed in fp32, then STORED AS BF16
q = round(W / scale.to(bf16)).clamp(-128, 127)
Casting the scale to BF16 before dividing is load-bearing — it is what reproduces the
parent's statistics (row maxima usually 127 but often 120–126, and -128 present).
Dividing by an fp32 scale and casting afterwards yields subtly different weights.
Activations are unchanged: 8-bit per-token dynamic, computed at inference.
No calibration data is used or needed — this step is a pure weight transform.quantize_gdn_int8.py in this repo reproduces it from the parent checkpoint exactly.
Max relative weight error introduced: 0.39%.
Evaluation — and its limits
Measured against the same checkpoint with GDN projections left in BF16, same harness,
same hardware.
Perplexity: +2.3% (token-weighted over 4,437 tokens)
| corpus | tokens | W8A8 | this |
|---|---|---|---|
| prose | 317 | 4.6624 | 4.3646 |
| source A | 2,347 | 2.3573 | 2.4441 |
| source B | 1,773 | 1.9029 | 1.9409 |
Long-context recall: 9/9, identical to the unquantized-GDN control
| prompt tokens | depths 10/50/90% | W8A8 | this |
|---|---|---|---|
| 14,147 | 3/3 | 3/3 | |
| 56,788 | 3/3 | 3/3 | |
| 107,337 | 3/3 | 3/3 |
The needle test was run against both checkpoints on purpose: this hardware declines the
fast paged-attention kernel, so a long-context failure could have been the attention path
rather than the weights. Running both makes the result attributable.
What was NOT evaluated. No standard benchmark suite (MMLU, GSM8K, HumanEval, etc).
No multi-turn or long-generation testing. No coding or math evaluation. The perplexity
probe is three passages, two of them Python source from one project. The needle test is a
single fact. Given that this deliberately quantizes a path the usual recipe protects, and
that GDN is recurrent so errors can compound in ways short probes do not reveal,
treat these numbers as a smoke test, not a quality claim. If you evaluate it properly,
please open a discussion with what you find.
Running it
vLLM on ROCm, single MI210:
vllm serve <this-model> \
--tensor-parallel-size 1 \
--max-model-len 131072 \
--gpu-memory-utilization 0.90 \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'
with VLLM_ROCM_USE_AITER=1. Notes for gfx90a specifically:
VLLM_ROCM_USE_AITER=1needs an image whose AITER carries gfx90a code objects, or
the INT8 GEMM silently falls back to a generic Triton kernel at roughly a third of the
rate. Confirm withgrep 'Selected.*ScaledMMLinearKernel'in the server log — you wantAiterInt8ScaledMMLinearKernel, notTritonInt8ScaledMMLinearKernel.- On a host with GPUs of more than one architecture, pass only the gfx90a render
nodes (--device=/dev/dri/renderD<N>, not all of/dev/dri). vLLM resolves the
architecture once at import from amdsmi's physical device 0 and ignoresHIP_VISIBLE_DEVICES/ROCR_VISIBLE_DEVICES, so a foreign card can switch every gfx9
path off silently. That alone was worth 2.4x here. cudagraph_mode: FULL_DECODE_ONLYmatters — vLLM otherwise drops to PIECEWISE under
speculative decoding.num_speculative_tokens: 1; N=2 measured worse on this part.
Tooling for gfx90a: davetha/mi210-vllm.
Provenance
Qwen/Qwen3.8-27B → Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16 (abliteration) →
W8A8 INT8 (llm-compressor, weights RTN + dynamic per-token activations) → this
(GDN projections additionally INT8, quantize_gdn_int8.py).