license: apache-2.0
base_model: orcarouter/Qwen3.8-27B-Uncensored
base_model_relation: quantized
pipeline_tag: text-generation
library_name: transformers
tags:
- nvfp4
- modelopt
- sglang
- dflash2
- speculative-decoding
- dgx-spark
- gb10
Qwen3.8-27B-Uncensored-NVFP4-W4A4
NVFP4 quantization of orcarouter/Qwen3.8-27B-Uncensored,
built to be served with DFlash2 speculative decoding on a single bandwidth-limited
Blackwell GPU. The layout matches RadixArk/Qwen3.8-27B-NVFP4 (revision 52d1adc).
21 GB on disk.
Why this layout
DFlash2 drafts a block of tokens per step and the target verifies them. That makes decode
even more bandwidth-bound than usual, and it adds one specific cost: the drafter's
candidate selector reads the target's lm_head a second time every step, on top of
the verify pass. A BF16 head is 2.5 GB; NVFP4 is 0.7 GB. Paid twice per step, that
difference is worth real throughput on a memory-bandwidth-limited card.
So everything that can be NVFP4 is NVFP4 — all 64 layers' MLP projections and the output
head — with FP8 reserved for the attention and linear-attention projections.
Layout
| Modules | Format |
|---|---|
All 64 layers' MLP gate_proj / up_proj / down_proj, plus lm_head |
NVFP4 W4A4, group 16 (193 modules) |
Self-attention q/k/v/o_proj; Gated DeltaNet in_proj_qkv / in_proj_z / out_proj |
FP8 W8A8 (208 modules) |
| KV cache | FP8 scales |
Vision tower, MTP head, conv1d, in_proj_a/b, embeddings, norms |
BF16 |
hf_quant_config.json's per-layer map is byte-identical to RadixArk's pinned revision.
The MTP head (15 tensors) is preserved unquantized and listed in exclude_modules, so
MTP speculative decoding remains available as a fallback if you'd rather not run DFlash2.
Serving with DFlash2
Pairs with the z-lab/Qwen3.8-27B-DFlash2
drafter (BF16, ~3.85 GB). SGLang flags:
--speculative-algorithm DFLASH \
--speculative-draft-model-path z-lab/Qwen3.8-27B-DFlash2 \
--speculative-num-draft-tokens 8 \
--speculative-draft-model-quantization unquant
Keep the drafter unquantized — a quantized drafter loses more in acceptance than it saves
in bandwidth, and quantized-drafter support is still incomplete upstream.
DFlash2 support is recent. At time of writing it requires a build carrying SGLang PR
#35371 plus a fix allowing the candidate selector to run through a quantized lm_head
— which this checkpoint has. The failure mode when an engine can't is usually a crash or
silently disabled speculation, not wrong output. Speculative decoding is lossless: the
target verifies every drafted token, so acceptance affects speed only, never output.
The lm_head trade-off
A quantized lm_head shifts the candidate selector's top-k slightly, which can lower
draft acceptance. RadixArk moved their own checkpoint's head back to BF16 for that reason
— a sensible call on a GB300, where 3.4 GB/step is noise. On a GB10 that same 3.4 GB is
about 15% of the step, and the NVFP4-head layout came out faster end-to-end despite any
acceptance cost. Which side you land on depends on your memory bandwidth. If you want a
BF16-head variant, the recipe in this repo has a comment showing the two-line change.
Reproduction
- NVIDIA Model Optimizer, commit in
modelopt-commit.txt - Recipe:
orca-nvfp4-w4a4-fp8attn.yaml(in this repo) - Calibration:
cnn_dailymail3.0.0, 1024 samples, seq len 512, batch 8, max-calibration - Per-module result:
quant_summary.txt
Caveats
- Requires Blackwell (NVFP4 tensor cores). Built and tested on GB10 (sm_121).
- This is a quantization of an abliterated model: the base has had refusal behaviour
removed and will not decline requests the original Qwen3.8-27B would. Safety
characteristics come from the base model, not from this quantization. Use accordingly.
License and attribution
Apache 2.0, inherited from Qwen/Qwen3.8-27B via orcarouter/Qwen3.8-27B-Uncensored.
Layout follows RadixArk/Qwen3.8-27B-NVFP4. Drafter by z-lab.