license: apache-2.0
base_model: orcarouter/Qwen3.8-Flash-Next-Uncensored
tags:

• mlx
• omlx
• awq
• quantized
• qwen4_exp
• moe
• mtp
• vision-language
  pipeline_tag: image-text-to-text

Qwen3.8-Flash-Next-Uncensored — MLX mixed-precision 4.77 bpw

Mixed-precision AWQ/affine quantization of
orcarouter/Qwen3.8-Flash-Next-Uncensored
(qwen4_exp, 180B) written to fit an M5 Max 128 GB box at full 262k context.

107.37 GB · BPW 4.77 · 180.00B params · 3.35× vs bf16 (360.00 GB) — routed experts 4-bit gs64,
n-gram PLE table 4-bit gs32, GDN in_proj 6-bit gs64 / out_proj 5-bit gs64, attention q/k/v +
embeddings + lm_head + shared experts + vision 8-bit, o_proj and the sparse-attention indexer
6-bit gs64, routers / hyper-connections / norms / SSM state exact bf16.

How it was made

• Hand-written per-module {bits, group_size} map (mode: affine), applied by monkey-patching
  omlx.oq.universal_quant_predicate and running quantize_oq_streaming tensor-by-tensor
  (~3–4 GB host RAM for a 360 GB source).
• 0 fallbacks across 2,134 predicate decisions — the map covers the model completely; nothing
  silently took oMLX defaults.
• Bit placement follows weight mass vs. sensitivity: 68 GB of 4-bit experts (≈half of trunk prefill
  FLOPs, and the NAX gs64 tensor-op path), 8-bit everywhere small and quality-critical, 5–6 bits on the
  recurrent-state path as an explicit quality-over-speed trade.
• The n-gram table is 4-bit gs32 because it has to be — PLE rows are 160 wide (8 heads × 20), and
  64 does not divide 160. Same class of constraint as the 27 vision linear_fc2, which stay bf16
  because input dim 4304 admits no supported group size.
• MTP head (2.61B params / 1.53 GB) produced by a text-only pass and grafted in as
  model-mtp.safetensors (58 tensors), since the stock validator rejects a qwen4_exp recipient.
• Byte split as measured from the checkpoint: 91.20 GB packed quants + 14.38 GB bf16 scales/biases
  • 1.79 GB kept exact. Tokenizer, chat template and configs are the source's own files.
• Not re-calibrated beyond the quantizer's own pass — no training, fine-tuning, or weight editing.

Requires oMLX

Needs oMLX ≥ 0.6.3rc3 (model_type: qwen4_exp, VL engine, pre-quantization sanitize). Two things are
engine-level model settings, not request parameters: mtp_enabled (Lightning MTP decode — the
"enable_mtp" request field is ignored) and PLE mode.

Set PLE to mode="mmap". With the n-gram table resident the build loads in 94 s and holds
107.6 GB hot, which macOS killed under memory pressure on a 128 GB desktop. mmap loads in ~11 s and
resides at 75.7 GB, which is what has been serving stably.

Measured on M5 Max 128 GB / oMLX 0.6.4: 30.7 tok/s decode aggregate (25,505 tokens over 30 agentic
tool-calling turns, 12.5k → 79k context, 0 errors), 91.7% prompt-cache hits. Trunk KV is only 24 KB/token
(12 of 48 layers cache KV) — 6.4 GB at 262k, 3.2 GB at 128k. Speculative-prefill token pruning does not
apply here: GDN state needs every token and pruning breaks n-gram hash adjacency.