license: other
license_name: qwen-community-1.0
license_link: LICENSE
base_model: lychee888/Qwen3.8-Flash-Next-Uncensored-NVFP4-FP8PLE
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: vllm
tags:

• qwen
• qwen3.8
• qwen4
• flash-next
• moe
• nvfp4
• fp8
• compressed-tensors
• mtp
• speculative-decoding
• abliterated
• uncensored
• vllm
• vision-language
• function-calling
• reasoning

Qwen3.8-Flash-Next-Uncensored-NVFP4-FP8PLE-OfficialMTP

lychee888/Qwen3.8-Flash-Next-Uncensored-NVFP4-FP8PLE
with one file replaced: model-mtp.safetensors now holds the official MTP head from
Qwen/Qwen3.8-Flash-Next instead of the abliterated one.
The main model weights are unchanged, byte for byte.

The upstream MTP head was abliterated together with the model. That lowered speculative-decoding
acceptance without doing anything for the refusal removal, because the head only proposes draft
tokens and the main model verifies them. With the official head, decode on vLLM is +9% at short
context and +18% at 250k tokens (see Benchmarks).

This repo also includes a ready-to-run vLLM setup for a single 96 GB GPU, with the full 262k context
(Quick start).

[!WARNING]
The safety alignment has been removed (abliteration). The model follows harmful, unethical or
illegal requests that the original Qwen3.8-Flash-Next refuses. It is intended for research: interpretability,
refusal-mechanism studies, red-teaming, robustness evaluation. You are responsible for how you use it
and for what it generates. Add your own moderation layer before exposing it to anyone.

Provenance

The chain from the base model, with each step and the exact revision used:

Qwen/Qwen3.8-Flash-Next                                   BF16 base, official MTP head
│   rev de4b8e4d43b917e7706784d8bb445c9af86a3540
│
└─ orcarouter/Qwen3.8-Flash-Next-Uncensored               abliteration (refusal direction removed), BF16
   │   rev 8336e613ea508b13c2159bd0f68965d97a606b95        MTP head abliterated as well
   │
   └─ orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4      compressed-tensors: NVFP4 experts (weight-only),
      │   weights as of c1209bda15a6 (2026-08-29)          FP8 per-channel attention / linear-attn / shared expert,
      │                                                    PLE n-gram table kept BF16 (170.9 GiB)
      │
      └─ lychee888/Qwen3.8-Flash-Next-Uncensored-NVFP4-FP8PLE
         │   rev 1b3c3cedc14fb3bee4294dea5e3515af74b8268a  PLE n-gram table → FP8 e4m3, one global scale
         │                                                (123.3 GiB)
         │
         └─ this repo                                      model-mtp.safetensors ← official Qwen MTP head
                                                           (from Qwen/Qwen3.8-Flash-Next @ de4b8e4)

Links: Qwen/Qwen3.8-Flash-Next ·
orcarouter/Qwen3.8-Flash-Next-Uncensored ·
orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4 ·
lychee888/Qwen3.8-Flash-Next-Uncensored-NVFP4-FP8PLE

What changed vs lychee888

The two MTP heads differ in exactly 3 of 31 tensors: mlp.experts.down_proj,
mlp.shared_expert.down_proj and self_attn.o_proj. Abliteration edits these tensors: it writes into the residual
stream, so these are the matrices it orthogonalizes. The other 28 tensors are identical.
unsloth/Qwen3.8-Flash-Next is a byte-identical mirror of the base repo, so this is also the head
inside unsloth's MTP GGUF drafts.

The MTP head only proposes draft tokens, and the main model verifies every one of them
(speculative decoding keeps the main model's output distribution). A different head therefore does not
change what the model says. It only changes how many drafts get accepted, which affects speed. Refusal
behaviour is that of the upstream model.

Quantization (unchanged from upstream)

config.json is authoritative. recipe.yaml is kept from upstream for provenance, but it predates
orcarouter's lm_head fix (c1209bd): it still lists lm_head in the FP8 group, while in the weights
lm_head is BF16.

Benchmarks

All numbers below were measured by me on one RTX PRO 6000 Blackwell Workstation (96 GB), Ryzen 9 9950X3D, 125 GB DDR5,
Docker on WSL2. vLLM v0.30.0 with the settings from serve/, one request at a time, MTP with 3 speculative tokens.
Decode is measured on real prose (Moby-Dick) sampled at temperature 1.0, median of 3 runs.
Prefill uses a synthetic prompt; prefill throughput does not depend on the content.
Batched throughput was not measured.

MTP head (the only difference between this repo and lychee888):

Throughput of this repo on vLLM, compared with llama.cpp running a Q4_K_M GGUF of the same fine-tune on the same machine:

On vLLM, decode speed does not drop with context depth. In these runs it was slightly higher at
250k, together with MTP acceptance (56% vs 50%).

Quality was not re-evaluated. The main-model weights are byte-identical to lychee888, so their evaluations
and orcarouter's still apply.

Quick start (vLLM, single 96 GB GPU)

The serve/ folder has a working docker-compose.yml. The repo is gated: first accept the terms
on this page, then use a Hugging Face token from that account.

export HF_TOKEN=hf_...   # the account that accepted the terms
# get serve/ (a few KB; the weights are pulled by vLLM on first start)
hf download STL1te/Qwen3.8-Flash-Next-Uncensored-NVFP4-FP8PLE-OfficialMTP --include "serve/*" --local-dir .
cd serve
docker compose up -d && docker compose logs -f   # first start downloads 123 GiB

The OpenAI-compatible API is then at http://127.0.0.1:8000/v1, with the model name
qwen3.8-flash-next-uncensored-nvfp4. Reasoning goes into reasoning_content, and tool calls are parsed
(qwen3_xml).

Requirements

• vLLM ≥ 0.30.0. Older releases cannot load the FP8 PLE table of a compressed-tensors checkpoint.
• GPU with ~92 GB free. About 76 GiB of weights stay on the GPU, including the MTP head; the rest is KV cache.
  Tested only on an RTX PRO 6000 Blackwell (SM120). The experts run through vLLM's Marlin kernel
  (weight-only FP4), which does not depend on Blackwell FP4 tensor cores. Other GPUs may therefore work, but I have not tested any.
• ~64 GB of free host RAM. vLLM keeps the 47.6 GiB PLE table in pinned host memory, and torch rounds
  pinned blocks up to a power of two.
• ~124 GiB of disk space for the weights.

Why each setting is there

• serve/qsa_indexer.py is vLLM PR #57105 (not yet merged),
  backported onto v0.30.0 and bind-mounted over the original file. Without it, the QSA indexer allocates a larger
  logits buffer for every prefill chunk (#56457), about 14 GB
  of garbage per long prompt. On Linux that ends in an OOM. On WSL2 the prompt silently stalls at ~170k tokens.
  Drop the mount once a vLLM release includes #57105. It only matters for prompts longer than ~100k tokens.
• --gpu-memory-utilization 0.93: with the patch, 0.92 leaves 7.56 GiB for KV cache against 7.57 GiB needed
  for 262,144 tokens. At 0.93 the KV pool holds ~291k tokens.
• --max-num-seqs 1: the settings are for a single user who gets the full context. For batched serving,
  raise it and lower --max-model-len accordingly.
• VLLM_WSL2_ENABLE_PIN_MEMORY=1 is needed on WSL2 only, and does nothing elsewhere. On WSL2, do not set
  PYTORCH_CUDA_ALLOC_CONF=pinned_max_round_threshold_mb. It makes the pinned PLE block its exact size,
  and a pinned block between ~38 and 64 GiB costs (size − 37.8 GiB) of VRAM there, which is enough to stop 262k from fitting.
• VLLM_LOGGING_LEVEL=DEBUG breaks CUDA graph capture on this model. Leave it off.

License

The base model Qwen/Qwen3.8-Flash-Next is released under the
Qwen Community License 1.0, and its text is included here as LICENSE. The license requires the copyright and
permission notice to accompany derivative works. It also places conditions on very large commercial deployments and on
"Model as a Service" / "AI Work Assistant" businesses; read it before any commercial use.
The intermediate repos are labelled Apache-2.0. The base model's terms still apply to derivatives, which is why this
repo carries the base license.

Credits

• Qwen: base model and MTP head
• orcarouter: abliteration and NVFP4 quantization
• lychee888: FP8 PLE table