language: en
library_name: vLLM
base_model: orcarouter/Qwen3.8-Flash-Next-Uncensored
pipeline_tag: text-generation
license: apache-2.0
tags: [nvfp4, awq, gptq, quantization, moe]
Qwen3.8 Flash Next — NVFP4 (AWQ + GPTQ, calibrated)
Calibrated 4-bit weights of orcarouter/Qwen3.8-Flash-Next-Uncensored: routed experts in NVFP4
(W4A4, AWQ smoothing + GPTQ error compensation on ~1K calibration
sequences), attention / GDN / shared-expert in FP8 per-output-channel;
embeddings, lm_head, norms, router gates and the PLE table kept in BF16.
Includes the MTP head (model-mtp.safetensors).
Benchmarks
Measured against the BF16 base model on 328,800 positions of
self-distilled thinking-mode output (8 documents: 2 multi-turn agentic
coding sessions with tool loops, 6 plain reasoning/math/writing seeds,
all generated to natural stop), teacher-forced top-1/top-2 at every
position. 8xRTX Pro 6000, TP8 + expert parallel, vLLM 0.29.0, BF16 KV
(see Serving notes). Agreement = raw argmax match vs the BF16 base;
buckets split by the BF16 top1-top2 logprob margin. Disagreement % shown
(lower is better).
| checkpoint | top-1 | near-tie | moderate | confident | certain |
|---|---|---|---|---|---|
| orcarouter community NVFP4 | 93.90% | 39.82 | 12.34 | 2.60 | 0.15 |
| this model (NVFP4 + AWQ + GPTQ) | 93.93% | 39.67 | 12.31 | 2.59 | 0.12 |
Per-category disagreement vs BF16 (tokens scored):
| category | tokens | this model | community NVFP4 |
|---|---|---|---|
| coding (agentic, tool loops) | 200,682 | 7.76% | 7.79% |
| math | 53,166 | 3.98% | 4.05% |
| reasoning | 63,769 | 2.39% | 2.38% |
| writing | 11,175 | 6.67% | 6.95% |
Paired per-position logprob test on the BF16-preferred token: this model
carries +0.0019 nats over the community quant (SE 0.0005, t=+4.1,
321,736 paired positions). Residual disagreement concentrates in BF16
near-ties (whitespace width, table digits, synonym flips) — sampled
splits show no reasoning or code defects on either side.
Serving
Tested with vLLM >= 0.29 (native qwen4_exp support), TP8 with--enable-expert-parallel on 8xRTX Pro 6000 (tree is ~180 GB):
vllm serve lambsea/Qwen3.8-Flash-Next-Uncensored-NVFP4-AWQ-GPTQ \
--tensor-parallel-size 8 --trust-remote-code \
--enable-expert-parallel --enable-auto-tool-choice \
--tool-call-parser qwen3_coder
Notes:
- KV cache must be BF16 for now: vLLM's QSA layers for this
architecture raiseQwen4Exp QSA does not support KV quantization. The
shippedconfig.jsondeclares nokv_cache_scheme. - The quantization manifest targets use the
model.language_model.layers.*
prefix (VLM-wrapper naming); the checkpoint loads through theQwen4ExpForConditionalGenerationpath even for text-only serving, sopreprocessor_config.jsonmust stay in the repo. - FP8 tensors carry per-output-channel
weight_scaleof shape[out_features, 1]; NVFP4 experts carry block-16weight_scale
(F8_E4M3, in global-scaled space) plus a quant-side scalarweight_global_scale— i.e. the kernel dequant multiplier is1 / weight_global_scale, the standard checkpoint convention.
Provenance
Quantized 2026-09 on 8xH100 (8-GPU DDP calibration, compressed-tensors
disk-offload pipeline), exported to a serving tree, repaired scales
(FP8 per-channel re-quantized from the linearized BF16 source; NVFP4
global scales normalized to the quant-side convention), and validated
end-to-end by the benchmark above. All measurements, corpus, and the
BF16 ground-truth score files are reproducible from the published run
profile.