license: apache-2.0
base_model: JonathanColetti/Qwen3.8-27B-Uncensored
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- qwen3_5
- qwen3.8
- fp8
- w8a8
- mtp
- speculative-decoding
- vision
- multimodal
- abliterated
- uncensored
- vllm
language: - en
- zh
- th
Qwen3.8-27B-Uncensored-FP8-MTP
FP8 (block 128×128, E4M3) quantization of an abliterated Qwen3.8-27B, with the
vision tower and the working MTP speculative-decoding head both preserved.
What "abliterated" means here. The base model has had its refusal direction removed —
it will not decline requests the way the official Qwen3.8-27B does. That modification comes
from the base model, not from this repository; this repo contributes the FP8 quantization
only. The base is Apache-2.0 and publicly available. Use accordingly.
Why this build
Surveying 148 abliterated/uncensored Qwen3.8-27B repositories in August 2026 turned up exactly
one FP8 build, and it was access-gated. Everything else was GGUF, NVFP4 (Blackwell-only), AWQ,
or unquantized BF16. On Hopper hardware — where FP8 has native tensor-core support — that left
no ungated option.
Only a handful of builds in any format preserve the MTP head, and getting it to actually
work under vLLM required the checkpoint to be in the native fp8 format rather thancompressed-tensors (see below).
Measured against the official build
Same hardware (H100 47GB vGPU), same vLLM 0.27.1, same flags.
Qwen/Qwen3.8-27B-FP8 |
this build | |
|---|---|---|
| Thai extraction, exact codepoint (temp 0, 4 runs) | 16/16 | 16/16 |
| Coding tasks — generated code executed against assertions | 4/5 | 4/5 |
| Vision: Thai table, activity names read exactly | 4/4 | 4/4 |
| Vision: numeric cells | correct | correct |
| MTP acceptance rate | 58% | 52% |
| tok/s @ concurrency 1 | 131 | 132 |
| tok/s @ concurrency 8 | 816 | 813 |
| tok/s @ concurrency 16 | — | 1,373 |
The one coding task both models fail is the same one, for the same reason: the model spends
its whole token budget reasoning and never emits an answer.
Hardware requirements
FP8 needs tensor-core support to be worth anything. On architectures without it, vLLM either
refuses the checkpoint or falls back to a dequantize-then-multiply path that is slower than
a 4-bit AWQ build.
| Architecture | Examples | Compute cap. | FP8 tensor cores | Runs this model |
|---|---|---|---|---|
| Volta | V100 | 7.0 | ✗ | no — also lacks BF16 tensor cores, and vLLM has dropped Volta |
| Turing | RTX 20xx, T4 | 7.5 | ✗ | no |
| Ampere | RTX 30xx, A100, A40, A6000 | 8.0 / 8.6 | ✗ | see note below |
| Ada Lovelace | RTX 4090, L4, L40S | 8.9 | ✓ | yes, if VRAM allows |
| Hopper | H100, H200, H20 | 9.0 | ✓ | yes — tested here |
| Blackwell | RTX 50xx, B100/B200, GB200 | 10.0+ | ✓ (plus FP4) | yes |
On Ampere (A40, A100, A6000). These have the VRAM — an A40 has 48 GB, the same as the
H100 47C this was tested on — but no FP8 tensor cores. vLLM can fall back toFp8MarlinLinearMethod, which keeps weights in FP8 and dequantizes inside the kernel, so
memory still works out. Whether that path accepts block-wise 128×128 scales (as opposed
to per-tensor or per-channel) is untested here — we had no Ampere card to try it on. If you
are on Ampere, an INT8 W8A16 or AWQ build of the same base is the safer choice.
VRAM
Weights occupy ~30 GB. The rest goes to KV cache, which decides how much context you can
actually serve.
| VRAM | Verdict | KV pool at --kv-cache-dtype fp8 |
|---|---|---|
| 24 GB (RTX 4090) | will not load on one card | — |
| 32 GB (V100 32GB) | no — wrong architecture entirely | — |
| 48 GB (A40) | VRAM is fine; FP8 path is not — see note above | — |
| 47–48 GB (H100 47C vGPU, L40S) | works | ~350k tokens — measured on H100 47C |
| 80 GB (H100/H200 80GB) | comfortable | ~600k+ tokens |
2× 24 GB with --tensor-parallel-size 2 |
works on Ada | splits weights ~15 GB per card |
At 47 GB the whole 262,144-token context fits for a single request with room for roughly one
more — Maximum concurrency for 262,144 tokens per request: 1.3x. Shorter contexts scale
proportionally: ~20 concurrent requests at 16k each.
Without --kv-cache-dtype fp8, KV for 262k context needs 16.17 GiB instead of ~8.5 GiB and
a 47 GB card will refuse to start.
Serving
vllm serve pramoths/Qwen3.8-27B-Uncensored-FP8-MTP \
--max-model-len 262144 \
--gpu-memory-utilization 0.94 \
--kv-cache-dtype fp8 \
--max-num-seqs 16 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder \
--enable-prefix-caching \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
--kv-cache-dtype fp8 is required for 262k context on a 47GB card — fp16 KV needs 16.17 GiB
and will refuse to start. --max-num-seqs must stay below the available Mamba state block
count; the default of 256 fails to boot.
Verify MTP is actually working — it can fail silently:
curl -s localhost:8000/metrics | grep spec_decode_num_accepted_tokens_total
A Mean acceptance length of 1.00 means every draft is being rejected. You are paying the
drafting cost for nothing.
What was and was not quantized
Follows Qwen/Qwen3.8-27B-FP8 exactly — the distinction is matmul versus control parameter,
not which component of the model:
- FP8:
q/k/v/o_projandgate/up/down_proj, in the language layers and in the MTP head. - BF16: LayerNorms, gates,
A_log,dt_bias,conv1d, and the gated-DeltaNetin_proj_a/in_proj_b— these govern recurrent state, where quantization error
accumulates rather than cancelling. - BF16: the entire vision tower,
lm_head,embed_tokens,mtp.fc.
Reproducing
Scripts, benchmarks, and a writeup of the five silent failure modes encountered along the way:
https://github.com/pramoth/qwen38-fp8-forge
Quantization needs no calibration data and no GPU — roughly two minutes of CPU.
Attribution
Qwen/Qwen3.8-27B— original model, Apache-2.0JonathanColetti/Qwen3.8-27B-Uncensored— abliterated baseQwen/Qwen3.8-27B-FP8— the quantization recipe this build copies