license: apache-2.0
base_model: Qwen/Qwen3.8-Flash-Next
tags:

• gguf
• qwen
• qwen3.8
• qwen4exp
• mixture-of-experts
• quantized
• mixed-precision
• uncensored
• strata
• xolotl
• consumer-gpu
  language:
• en
• zh
• code
  pipeline_tag: text-generation

Qwen3.8-Flash-Next · XQ mixed-tier, abliterated

A 125B-parameter MoE (6B active) running on one gaming PC: RTX 4090 (24 GB) + 90 GB RAM + NVMe,
at 152 tok/s decode and 4,285 tok/s prefill.

Runs on Xolotl, an engine built for exactly this model and
this class of machine, on top of Strata.

This is a refusal-abliterated build: a rank-2 refusal-direction edit was applied to the model
before quantisation. The edit touches the dense backbone; the routed experts are untouched.

What it is

Qwen3.8-Flash-Next is a sparse MoE with 48 layers and 512 routed experts per layer, plus a 51.2B
n-gram (PLE) table held off-accelerator. QwenLM describes the architecture as "an early preview of
the architecture used in Qwen4", and the GGUF's arch string is literally qwen4exp.

This is not a llama.cpp / Ollama / LM Studio GGUF. It uses per-expert quantisation types those
runtimes do not load — they expect one type per tensor, and here each layer's experts can carry
different formats. Use Xolotl.

How the bits are spent

Most of a MoE's parameters live in experts that any one token touches rarely, so a single global
bit-width is the wrong shape for this model. Instead:

• Every routed expert gets its own format. The gate/up projections and the down projection are
  chosen separately, per layer, from a tier ladder measured on real routed tokens.
• The down projection stays at a higher precision than gate/up. A down projection writes
  directly into the residual stream, so its error is not averaged away the way gate/up error is.
  This is also what preserves the abliteration edit, which lives in the down path.
• The dense backbone is Q8_0. On this file the Q8_0 emit is exactly identity (the source was
  already Q8_0, and amax/127 recovers the block scale bit-exactly), so the backbone is
  bit-identical to the parent at +2.4 GiB over the nearest lossy tier. That is a deliberate
  quality choice, not an oversight.
• The PLE table is a separate tier and is read from disk with O_DIRECT rather than held in
  RAM or VRAM — each row is read whole and its errors do not average out, so it gets a precision of
  its own.

The tier assignment is produced by a calibration-driven allocation. The allocation method, the
screening procedure and the tier ladder's derivation are not published.

Format mix in this file

Speed

Measured on this machine: Linux, RTX 4090 24 GB, Ryzen 7 5700X, 90 GB DDR4-3200, model on a
3.5 GB/s NVMe. 16,383-token prompt, 1,024 generated tokens, greedy:

Decode is bound by RAM bandwidth once the expert arena is warm, so numbers scale with memory
speed more than with the GPU. Longer generations read higher than short ones; a 64-token reply
measures lower than this table.

Tags: xol-qfn.

Quality

Being measured — placeholders below will be filled in before this card is announced.

What is already established: the quantisation was validated by reading the output file (arch,
tensor count, per-tensor types) rather than by trusting the quantiser's exit code, and the
abliteration was verified to survive quantisation by measuring the component of the activation
change along the edit direction — 0.96–1.01 retained across early, middle and late layers. A
whole-tensor norm ratio is the wrong metric for that test and was not used.

Quick start

git clone https://github.com/Indras-Mirror/Xolotl.git xolotl && cd xolotl
bash tools/xolotl/fetch-llama.sh
cmake -S . -B build-xolotl -G Ninja -DCMAKE_BUILD_TYPE=Release \
      -DSTRATA_ENABLE_CUDA=ON -DSTRATA_GGML_CUDA=ON -DSTRATA_MMQ_KQUANTS=ON \
      -DCMAKE_CUDA_ARCHITECTURES=89 -DSTRATA_GGML_DIR=$PWD/third_party/llama.cpp
ninja -C build-xolotl strata

Then serve it with the config below (strata-xol-qfn.json, port 8099). hf download the model and
its pack directory first; the pack is what teaches the engine this model's per-expert layout.

The engine treats the experts as a cache: the hottest ones in VRAM, the rest pinned in RAM, with
STRATA_ROUTE_RESIDENT preferring whichever tier is already resident. That bias is why the decode
number above is possible on a 78.83 GiB model in 90 GB of RAM.

Credits

• Qwen for Qwen3.8-Flash-Next (Apache-2.0).
• mrfakename for the rank-2 refusal-direction edit style.
• Niko1221 for Strata, the tiered MoE engine this builds on.
• ggml / llama.cpp for the compute backend.
• Abliteration, quantisation, engine port and measurements: Indra's Mirror (Xolotl).

Support

If this has been useful and you want to help keep it going, you can
buy me a coffee.