base_model:

• lychee888/Qwen3.8-Flash-Next-Uncensored-NVFP4-FP8PLE
• orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4
  license: other
  license_name: qwen-community-license-1.0
  license_link: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE
  library_name: vllm
  tags:
• quantized
• fp8
• lm-head
• vllm
• qwen3.8

Qwen3.8-Flash-Next — FP8 output head (lm_head) delta

A drop-in FP8 (E4M3, per-channel) lm_head for
lychee888/Qwen3.8-Flash-Next-Uncensored-NVFP4-FP8PLE (and, by construction, the
orcarouter/Qwen3.8-Flash-Next-Uncensored-* family), plus the small vLLM patch
that lets vLLM load a quantized ParallelLMHead, the exact quantisation script,
and the benchmark scripts + raw results.

Result: lm_head 1.27 GiB -> 0.64 GiB, decode +6–20 % steps/s at one stream,
GSM8K identical, MMLU within 1.5 points (n=200 each).

The output head is a dense [248320, 2560] projection read on every target
decode step, so halving its bytes is a fixed per-step saving. It also projects the
drafter's logits, because this model's MTP head shares the main lm_head.weight.

Files

FP8/lm_head_fp8.safetensors   # lm_head.weight (F8_E4M3 [248320,2560]) + lm_head.weight_scale (BF16 [248320,1])
scripts/quantize_head_fp8.py  # reproduce the FP8 head from the base bf16 head (per-channel, scale = amax/448)
scripts/apply_fp8_head.py     # splice it into a local copy of the checkpoint (updates index + config.json)
vllm/clean-model.py           # patched vllm/models/qwen3_8_flash_next/nvidia/model.py
vllm/clean-mtp.py             # patched .../mtp.py
vllm/Dockerfile.fp8head-v1    # FROM qwen38-blazux-direct:20260909, COPY the two files
bench/                        # bench_ab.py, bench_downstream.py, bench_speed.py, needle.py, raw results

Method

lm_head.weight bf16 -> FP8 E4M3 with a per-output-channel scale
(scale = amax / 448, stored BF16), matching the compressed-tensors scheme
(naive-quantized, 8-bit, strategy=channel, symmetric) already used by 300
attention/shared layers in this checkpoint. Added lm_head + re:.*lm_head$
to that config group and removed it from ignore.

The head's vLLM module name is language_model.lm_head (not lm_head), which
is why the config target is a regex.

vLLM patch (2 files, 3 edits)

1. model.py main head:
   ParallelLMHead(..., quant_config=without_modelopt_fp4(self.quant_config))
2. mtp.py drafter head: ParallelLMHead(..., quant_config=self.quant_config)
3. a small helper _set_head_linear_attrs(head) that gives the quantized
   ParallelLMHead the linear attributes the FP8 W8A16 Humming kernel reads
   (output_partition_sizes, input_size, input_size_per_partition,
   has_bias, output_size).

Stock vLLM fails on a 4/8-bit ParallelLMHead: the compressed-tensors scheme
creates the parameters (and the loader reads lm_head.weight_scale) but
process_weights_after_loading then needs those linear attrs.

Capability (n=200 each, enable_thinking=false, temperature 0)

Differences are 0–3 questions, inside the ±2 % sampling band.

Two exact, head-only measurements:

• weight error: relative Frobenius 2.88 %, spectral ‖ΔW‖/‖W‖ = 0.267 %;
• head A/B on 11,476 real final-layer hidden states (6,000 sampled):
  top-1 agreement 95.12 %, top-5 overlap 0.959, mean KL(bf16‖fp8) 0.0016,
  and every top-1 flip sat at a near-tie (top1−top2 gap median 0.039 vs
  1.733 overall).

Speed

Decode, single stream, warm, real prompts (3 reps):

An earlier warm boot of the same config measured up to +20 % steps/s on code.
The saving is a fixed per-step cost, so it is largest at c=1 and washes out by
c=4–8. Speculative acceptance is unchanged (code 87.8 / prose 75.5 / zh 66.7).

Prefill is unaffected (the head is applied to the last token only):
802 / 1327 / 1608 / 1503 / 1412 tok/s at 1k / 4k / 16k / 64k / 128k tokens.
Needle retrieval at 13k and 53k tokens: correct.

Reproduce

# 1) quantize the head (bf16 -> fp8 e4m3 per-channel)
python scripts/quantize_head_fp8.py --model-dir /path/to/base --out lm_head_fp8.safetensors

# 2) apply to a local copy (writes model-00017, updates index + config.json; backs up first)
python scripts/apply_fp8_head.py --model-dir /path/to/base --head lm_head_fp8.safetensors

# 3) build the patched image
docker build -f vllm/Dockerfile.fp8head-v1 -t qwen38-fp8head:v1 .

# 4) serve (same flags as the base stack)
vllm serve <model-dir> --kv-cache-dtype fp8 --async-scheduling \
  --kv-cache-memory-bytes 19327352832 --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Limitations

• Measured on one DGX Spark (GB10), vLLM v0.1.dev20073+g8e685d198, fp8 KV,
  MTP=2, --kv-cache-memory-bytes 18 GiB. Absolute tok/s are
  hardware/content dependent; the relative head saving is what transfers.
• Capability numbers are GSM8K/MMLU subsets (n=200), not full suites.
• A ~5 % argmax flip rate at near-ties is inherent to any head quantisation;
  downstream quality was unchanged within sampling error here, but this is not a
  proof of zero degradation on every task.
• The mkldnn-style naive-quantized scheme is per-channel; fine-grained
  block-128 FP8 was intentionally not used (different quant_method, and
  mixing it into a compressed-tensors checkpoint is not supported).

Attribution / license

Base weights: lychee888/Qwen3.8-Flash-Next-Uncensored-NVFP4-FP8PLE, derived from
orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4 and ultimately
Qwen/Qwen3.8-Flash-Next. The Qwen Community License 1.0 and the base
model's terms apply to any redistribution of these derivatives. The vLLM patch
is Apache-2.0 (vLLM), on top of NVIDIA's container image.