base_model:

• local-inference-lab/Qwen3.8-Flash-Next-NVFP4
• huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated-GGUF
  library_name: transformers
  pipeline_tag: image-text-to-text
  license: other
  license_name: qwen-community-1.0
  license_link: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/LICENSE
  tags:
• quantized
• nvfp4
• mxfp8
• modelopt
• abliterated
• sglang
• conversational
• multimodal

Huihui-Qwen3.8-Flash-Next-abliterated-NVFP4-QAD

Base:

• local-inference-lab/Qwen3.8-Flash-Next-NVFP4
• huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated-GGUF

Conversion: DQ(LIL QAD) + DQ(Huihui Q8) - DQ(Unsloth Q8), followed by BF16 rounding and ModelOpt requantization. Original LIL activation scales are retained without new activation calibration. GGUF residuals and requantization error remain; equivalence to Huihui BF16 and preservation of LIL benchmark scores are not claimed. See transfer-manifest.json.

Validated on DGX Spark TP1 with my SGLang: See runtime-qualification.json.

Recipe example:

MODEL_ID=edp1096/Huihui-Qwen3.8-Flash-Next-abliterated-NVFP4-QAD
MODEL_DIR="$HOME/.cache/huggingface/$MODEL_ID"
hf download "$MODEL_ID" --local-dir "$MODEL_DIR"

docker run -d --name qwen38-qad --gpus all --init --ipc=host \
  --shm-size=16g --ulimit memlock=-1 --security-opt seccomp=unconfined \
  -p 8000:30000 \
  -v "$MODEL_DIR:/model:ro" \
  -v "$HOME/.cache/qwen38-qad:/root/.cache/sglang" \
  -e SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 \
  -e SGLANG_QAD_B12X_GDN=1 -e SGLANG_QAD_PLE_IO_URING=1 \
  -e SGLANG_QWEN4_PLE_FILE_RSS_BUDGET_GB=0.5 \
  -e SPARKTALK_FLASH_NEXT_DRAFT_VOCAB=off \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  --entrypoint python3 ghcr.io/edp1096/sglang-qwen38-qad:sm121-v4 \
  /opt/sparktalk-flash-next/launch.py \
  --model-path /model --served-model-name "$MODEL_ID" \
  --host 0.0.0.0 --port 30000 --tp-size 1 \
  --context-length 1048576 --max-total-tokens 1048576 \
  --mem-fraction-static 0.85 --max-running-requests 1 \
  --chunked-prefill-size 4096 --page-size 64 --cuda-graph-max-bs-decode 1 \
  --quantization modelopt_mixed --kv-cache-dtype fp8_e4m3 \
  --prefill-attention-backend triton --decode-attention-backend trtllm_mha \
  --moe-runner-backend flashinfer_cutlass --fp4-gemm-backend flashinfer_cutlass \
  --disable-shared-experts-fusion --disable-flashinfer-autotune \
  --ple-offload-embedding --ple-offload-backend file \
  --ple-offload-dir /root/.cache/sglang/ple --weight-loader-drop-cache-after-load \
  --mamba-radix-cache-strategy extra_buffer --max-mamba-cache-size 8 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder --mm-feature-transport cpu \
  --speculative-algorithm NEXTN --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --speculative-draft-model-quantization modelopt_mixed \
  --json-model-override-args '{"text_config":{"rope_parameters":{"mrope_interleaved":true,"mrope_section":[11,11,10],"rope_type":"yarn","rope_theta":10000000,"partial_rotary_factor":0.25,"factor":4.0,"original_max_position_embeddings":262144}}}'

Model Description

local-inference-lab/Qwen3.8-Flash-Next-NVFP4 is a mixed-precision model distilled from
Qwen/Qwen3.8-Flash-Next
using quantization-aware distillation (QAD). The student is trained against
the original BF16 teacher with quantized weights in its forward pass,
learning to compensate for quantization error rather than relying on
post-training quantization alone.

The architecture is unchanged: 48 decoder layers, 512 routed experts per
layer with 10 active per token, hybrid Gated DeltaNet/Qwen Sparse Attention (QSA),
and n-gram embedding tables. Compression comes from lower-precision weights,
not fewer layers or experts. The checkpoint occupies approximately 98 GiB
on disk and is particularly suited to run on a single RTX 6000 (with PLE offload),
or a DGX Spark (with or without PLE offload).

What's quantized

NVFP4 stores 4-bit E2M1 values with FP8 block scales per 16 elements and an
FP32 global scale. MXFP8 stores 8-bit E4M3 values with power-of-two block
scales per 32 elements. Residual-mixing and PLE normalization weights are
trained in FP32 and exported in BF16. Shared-expert scalar gates remain
frozen in BF16.

Vision and MTP retain their weights. They are included in the release but were not part of text distillation.

Quantization-aware distillation

The BF16 teacher generates responses and supplies token-level probability
and hidden-state targets. The trainable NVFP4 and MXFP8 weights are quantized
and reconstructed for each forward pass. Gradients update
the underlying weights so their low-precision representations better match
the teacher.

The objective combines next-token probability matching using total
variation, hidden-state matching, and separate losses for end-of-response
and end-of-thinking behavior. These boundary losses target the teacher's
decisions to continue, finish reasoning, and stop responding.

Training uses the Quatrain distillation trainer: 2,500 trunk
updates followed by 1,500 joint-refinement updates, which also train the
n-gram embedding tables and simulate MXFP8 shared-expert weights. Attention
remains frozen, using its MXFP8 weights during joint refinement.

Training data

The distillation corpus contains 200,004,844 prompt-and-response tokens across
53,573 documents, with sequences up to 8,192 tokens. Prompts cover coding,
reasoning, general instruction following and multilingual conversations.
Assistant responses are generated by the original BF16 Qwen teacher using
its native chat template and thinking mode; responses from other models
are not used as training targets.

Activation calibration

Distillation uses BF16 activations. Separately, routed-expert activation
scales are calibrated over the 200M-token chat corpus and an additional
104M raw-text tokens, using natural routing. The raw-text corpus is used
for calibration, not gradient updates.

The serving configuration uses static NVFP4 activation quantization for
text routed experts and dynamic MXFP8 activation quantization for MXFP8
projections. PLE tables, MTP routed experts and vision FC2 use weight-only
NVFP4. Activation quantization is therefore separate from the weight
quantization simulated during distillation.

Requirements

Use a runtime that supports Qwen3.8-Flash-Next and this mixed NVFP4/MXFP8
ModelOpt layout, including NVFP4 PLE tables. The disk size is not a runtime
memory estimate: KV cache and runtime buffers require additional memory.
The tokenizer, chat template and vision assets are preserved from the
mixed-precision base checkpoint.

Evaluation

On GPQA Diamond, this distilled checkpoint scored higher than NVFP4 PTQ
while using 9% fewer tokens on average, 15% fewer at the median, and 26%
fewer at the 99th percentile (P99).