base_model: Qwen/Qwen3.8-Flash-Next
base_model_relation: quantized
library_name: mlx-serve
license: other
license_name: qwen-community-1.0
license_link: LICENSE
pipeline_tag: text-generation
tags:

• mlx
• mlx-serve
• qwen4_exp
• moe
• sparse-attention
• ngram-embedding

Qwen3.8-Flash-Next for mlx-serve (4-bit experts, 8-bit rest)

mlx-serve pack of Qwen/Qwen3.8-Flash-Next,
the Qwen4 preview architecture (model_type: qwen4_exp). About 63 GB
resident (128 GB Macs). Includes the MTP head and the vision tower (image and video input).

mlx-serve --model ddalcu/SamQuant--Qwen3.8-Flash-Next-Uncensored-MLX-Serve-Mix-C58 --serve

Measured on an M4 Max 128 GB (mlx-serve, first port, no tuning yet): ~67-69 GB
resident, decode 29-34 tok/s serial, prefill ~400 tok/s on a 25k-token prompt,
a needle at 24.8k tokens recovered with sparse attention engaged. The MTP head
loads and drafts (~1 accepted token per round) but its round cost is not yet
competitive with serial decode, so leave it off for now.

What is different about this model

This is not a Qwen3.5-style pack. Three things around the usual GDN + MoE trunk:

• Gated residual streams. The residual is 4 streams wide (4 x 2560). Every
  block reads a sigmoid-mixed average of the normalized streams and writes back
  through per-stream scalar gates. The final mixer replaces the usual final norm.
• N-gram embedding (51B parameters). A second embedding table indexed by
  hashed bigrams and trigrams of the token ids: 16 heads, each a prime-sized
  bucket space of ~20M rows, 160 dims per row, injected once before layer 1.
  It is a lookup, no compute, which is why Qwen quotes the model as 125B: the
  full checkpoint is 125B trunk + 51B n-gram + 4B MTP = 180B (360 GB bf16).
• Qwen Sparse Attention. Past 2048 tokens each attention layer only reads
  the 512 most relevant 4-token blocks per query (picked by a small indexer),
  plus the query's own partial block. Attention cost stays flat with context.
  Native 262k context.

How this pack stores the n-gram table

The 51B table is NOT in the safetensors shards. It is one merged 4-bit table
in ngram_table.bin (32.0 GB, safetensors format, .bin so nothing
mlx-loads it). mlx-serve mmaps the file and, per token, dequantizes the 16 rows
it needs on the CPU (16 x 80 bytes) and uploads only the resulting 2560-vector.
The table never becomes resident: its cost is page cache, which the OS evicts
as needed. That is the difference between this pack and mlx-lm style packs
that ship the table as 128 quantized tensors and load it onto the GPU
(+32 GB resident, ~107 GB total for a 4-bit pack).

Expected effect: decode speed unchanged (16 tiny reads against a ~20 ms step),
cold-cache prefill of very long prompts may pay up to ~1 s per 8k tokens of
random reads on the SSD, warm cache is free. No user-space cache is needed,
the page cache already is an LRU over exactly this access pattern.

Widths

Every (1 + w) RMSNorm has the +1 folded into the stored weight; depthwise
convs are transposed to MLX's [C, K, 1]; experts.gate_up_proj is split into
switch_mlp.gate_proj / up_proj. The vision tower ships dense bf16 in
model-vision.safetensors (~0.9 GB).

Serving notes

• Memory. ~63 GB resident plus KV cache. mlx-serve sizes the context to
  what fits; --kv-quant 8 halves the cache.
• MTP. The checkpoint's own 1-layer speculative head is loaded from the
  pack and works (--mtp or per-request "enable_mtp": true), but as of this
  build it decodes slower than serial. Default-off; a later mlx-serve release
  will flip it once the round cost is fixed.
• v1 limits in mlx-serve. One request at a time (no batched decode), no
  prefix-cache reuse between turns yet, PLD/DFlash speculation off (MTP is the
  speculative path). Very long prompts (past ~64k) want a smaller
  --prefill-chunk because the sparse-attention selection is built per chunk.
• Thinking is on by default ("enable_thinking": false turns it off).
  Tools use Qwen3.8's XML call format; mlx-serve parses and schema-coerces it.
• Images and video go through the Qwen3-VL-style tower (model.visual.*,
  dense bf16). MTP is declined on image turns (serial decode).

Conversion

tests/convert_qwen38_flash_next.py in the mlx-serve repo. It streams the
360 GB bf16 checkpoint shard by shard from the Hub (download, quantize, delete),
so it converts on a machine with ~150 GB free. The engine was validated against
HF transformers (trunk) and the vLLM/SGLang MTP math on a tiny random model
before the full conversion.