license: other
license_name: qwen-community-license-1.0
license_link: LICENSE
base_model: windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4
pipeline_tag: image-text-to-text
library_name: transformers
tags:

• nvfp4
• fp4
• fp8
• modelopt
• vllm
• abliterated
• uncensored
• qwen4_exp
• qwen3.8
• flash-next
• moe
• dgx-spark

Qwen3.8-Flash-Next-Abliterated-NVFP4-plefp8

windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4
with its 51.2B PLE n-gram embedding table stored in FP8 instead of BF16, and
its PLE tensors re-laid-out into the shard form vLLM's loader expects.

173.6 GiB → 125.9 GiB. Nothing else is touched: the NVFP4 body, the
abliteration, the vision tower and the MTP draft head are all copied through
byte-for-byte.

On 2× DGX Spark (TP2) that is 88.63 → 64.79 GiB per node, and the freed memory
goes almost entirely to KV cache.

Why the table dominates the file size

Qwen3.8-Flash-Next is 125B of body plus a 51.2B N-gram (PLE) embedding table
living in a single layer. The table is a hashed lookup, not a matmul, so every
NVFP4 quantizer excludes it from the 4-bit pass. What differs between published
NVFP4 builds is therefore mostly what dtype the excluded table is stored in:

NVFP4 shrinks the experts in every 4-bit build; a BF16 table hands the gain back.

What changed, exactly

1. PLE table split and renamed. Upstream ships the table as one consolidated
tensor:

model.language_model.layers.1.ple.ple_embedding.shard_.weight   BF16 [320001536, 160]  95.37 GiB

vLLM's ple_layer.load_weights only recognises ngram_embedding.shard_<int>.weight.
That name has neither the ngram_embedding. component nor a shard index, so it
falls through to AutoWeightsLoader, which has no parameter to bind it to. Here it
is split into split_ngram_parts = 128 shards of 2,500,012 rows
(320,001,536 / 128) named the way the loader expects — the same shape and naming
Qwen's own FP8 build uses.

2. Cast BF16 → float8_e4m3fn with a single global per-tensor scale, written
as ...ngram_embedding.weight_scale, bf16, shape (1,) — again matching Qwen's
FP8 build. The computed scale is 0.00019931793213, which is bit-identical to
the scale in Qwen's own FP8 checkpoint (abliteration barely moves the table's
dynamic range). Round-trip vs the source table: cosine 0.9997, max relative
error ~3.5%.

3. layer_types normalised. Upstream's config (emitted by transformers
5.16.0.dev0) labels the 12 QSA layers qwen_sparse_attention. vLLM rejects that
outright:

ValidationError: `layer_types` entries must be in (...) but got [... 'qwen_sparse_attention' ...]

The 12 layer positions are identical to Qwen's published config ([3, 7, 11 … 47]),
so they are renamed to full_attention. This fix is needed to serve the upstream
checkpoint too — it is not specific to the FP8 repack.

Does it need a patched vLLM?

Yes — one small patch. vLLM selects its FP8 PLE embedding method only when the
global quantization config is Fp8Config:

def _get_ple_embedding_quant_method(quant_config, prefix):
    if not isinstance(quant_config, Fp8Config):
        return None

This checkpoint's global config is ModelOptNvFp4Config (the body is NVFP4) with
*.ple.* in exclude_modules, so that returns None, the embedding is built in
BF16, and loading the F8_E4M3 shards fails. The patch decides from the on-disk
tensor dtype instead, which leaves BF16-table checkpoints on the unquantized path
unchanged.

The patch is included as patches/fix-qwen38fn-nvfp4-ple-fp8.sh (idempotent,
self-verifying) and is also carried in
eugr/spark-vllm-docker as
mods/fix-qwen38fn-nvfp4-ple-fp8.

Everything else — the NVFP4 body, vision tower, MTP head — loads on a stock
qwen4_exp-capable vLLM.

Measured: what the FP8 table costs

Both builds are identical except for the table's dtype (same weights, same shard
layout, same config fix), served on 2× DGX Spark, TP2, 262144 context:

Quality, first-token distribution over 80 harmless prompts, BF16 build as reference:

Long-generation degeneracy (20 runs each) and tool-calling (24 cases each) show no
regression: mean_distinct_3 0.9888 → 0.9894, max_ngram_run 1 → 1,
json_parse_rate 1.00 → 1.00.

The FP8 build picks the same first token as the BF16 build on every prompt
tested. For context, the NVFP4 body quantization both builds already carry is
roughly an order of magnitude lossier than this cast.

Serving (vLLM, 2× DGX Spark)

vllm serve <path-to-this-repo> \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.86 \
  --enable-prefix-caching \
  --no-enable-flashinfer-autotune \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL"}' \
  --tool-call-parser qwen3_coder --enable-auto-tool-choice \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Notes for GB10 / DGX Spark specifically:

• mode:0 disables Inductor. With compilation on, its autotuner tries to
  benchmark a kernel over the whole PLE table and asks for ~47.7 GiB in one
  allocation, which OOMs. cudagraph_mode: FULL still gives graph replay.
• VLLM_PLE_CPU_OFFLOAD=1 is single-node only (Unsupported settings: nnodes=2),
  so it cannot be used on a 2-node cluster. On unified memory it would not help with
  capacity anyway — host and device are the same physical pool.
• gpu_memory_utilization ceiling is lower than it looks: CUDA's free memory tracks
  the OS free column, not available, so page cache eats into the budget.

Reproducing

python3 repack_ple_fp8.py \
  --src <windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4 snapshot> \
  --dst <output dir>
# control build (split + rename, keep BF16):
python3 repack_ple_fp8.py --src <snapshot> --dst <dir> --keep-bf16

repack_ple_fp8.py is included here and in
eugr/spark-vllm-docker under
tools/ple-repack/. It streams the table in row slices, hardlinks every untouched
shard file, and rewrites the index — it never holds the 95 GiB table in memory.

Credits

• Qwen — Qwen3.8-Flash-Next, the base model, and
  the FP8 build whose PLE layout and scale convention this reproduces.
• windowsxp811203 — the abliteration
  and the NVFP4 conversion this repo is derived from. Their ARCHITECTURE.md
  (included) is the best documentation of qwen4_exp internals I have found, and
  their graft_nvfp4.py / fix_nvfp4_config.py document real failure modes in
  quantizing this architecture (silently dropped vision and MTP tensors). Both are
  intact here: 333 vision tensors and 31 MTP tensors verified present.
• RadixArk — their NVFP4 build demonstrated
  that an FP8 PLE table is viable, which is what prompted this repack.

Only the PLE table's dtype and layout differ from the upstream abliterated NVFP4
build. All credit for the model itself belongs upstream; any mistakes in the
repack are mine.

License

Qwen Community License 1.0, inherited from the base model. See LICENSE.