license: other
license_name: qwen-community-1.0
license_link: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE
base_model:
- Qwen/Qwen3.8-Flash-Next
- Blackfrost-AI/Qwen3.8-Flash-Next-DERISKED-BF16
base_model_relation: quantized
tags: - mtplx
- mlx
- apple-silicon
- qwen3.8-flash-next
- qwen4_exp
- speculative-decoding
- mtp
- abliterated
- uncensored
- derisked
- 4-bit
pipeline_tag: text-generation
library_name: mtplx
Qwen3.8-Flash-Next-Abliterated-MTPLX-4bit
An abliterated 4-bit build of Qwen3.8-Flash-Next for Apple Silicon, running on
MTPLX with multi-token-prediction speculative decoding.
✅ Runs natively on MTPLX 2.10.0 — no patches, no branches
As of MTPLX 2.10.0 (29 August 2026), Qwen3.8-Flash-Next is a native MTPLX
family and this build loads on the stock release:pip install mtplxEarlier revisions of this card required a patched branch. That is no longer
the case —qwen4_expsupport, the oQ mixed-precision path, and the native
MTP draft head all ship upstream now. This build carries anmtplx_runtime.jsonserving contract that 2.10.0 verifies on load
(arch_id: qwen4-next, turbo profile).
Lineage: Qwen/Qwen3.8-Flash-Next
→ Blackfrost-AI/Qwen3.8-Flash-Next-DERISKED-BF16
(abliterated) → Qwen3.8-Flash-Next-Abliterated-MLX-4bit
(oQ4e quantized) → this build (packaged for MTPLX).
The body and the MTP draft head are the same iMatrix-calibrated oQ4e weights as
the MLX build, passed through without requantization. Vision is not available
in this build (see Differences from the MLX build).
| This build: 4-bit | |
|---|---|
| Recommended: MTP depth 1 + resident n-gram | 50.1 tok/s |
| MTP depth 2 | 47.1 tok/s |
plain decoding (ar), resident n-gram |
33.9 tok/s |
| plain decoding, n-gram streamed from SSD | 30.0 tok/s |
| speculative speedup | 1.48x over plain decoding |
| MTP draft acceptance | 96.5% / 94.2% at depth 2 |
| Abliteration (upstream BF16, Qwentium persona) | 2.3% refusal on harmful prompts (7/300), 2.2% full suite (10/450) * |
Run this build with
--generation-mode mtp --depth 1andMTPLX_NGRAM_RESIDENT=1. Both levers matter: speculation is worth 1.48x
and resident n-gram arms MTPLX's pipelined decode lane for a further ~14%.
Earlier revisions of this card recommendedarand reported speculation as a
loss — that was a symptom of the RMSNorm defect fixed in this revision, not a
property of the model. See Speculative decoding.
Abliteration is measured by Blackfrost-AI on the parent
BF16 checkpoint
with the Blackfrost-AI "Qwentium" persona active. This 4-bit build has yet to be
re-measured after quantization, and ships with the stock Qwen persona rather
than the "Qwentium" persona, which is included as an optional swap; see
Personas. Prior work on
Qwen3.8-27B-Abliterated-MTPLX
showed Blackfrost-AI abliteration holding at the weight level with the persona
removed, with refusal rate of 8.7% at 1 attempt, 4.2% within 3 attempts, and
1.7% with the Blackfrost persona.
| download size | 108 GB (106 GB body + 1.5 GB MTP head) |
| resident when serving | ~106 GiB |
| base model | 125B parameters, ~6B active per token (MoE, 10 of 512 experts) |
| context | 262,144 tokens |
⚠️ Intended use and terms
Published for experimental AI research and AI-safety research only:
red-teaming, refusal and alignment research, interpretability, and quantization
or inference research.
It must not be used for anything illegal, in any jurisdiction, under any
circumstances. The removal of refusal behavior is a research property of the
artifact, not an invitation, a permission, or an endorsement. Nothing about
publishing this model condones or excuses illegal activity.
You are solely responsible for your use of it and for complying with
applicable laws and upstream licence terms. It ships with essentially no safety
refusals: assume it will comply with any instruction, including harmful ones.
Run it locally on hardware you control. Do not expose it as a public endpoint or
deploy it to untrusted users.
What changed for MTPLX 2.10.0
2.10.0 makes Qwen3.8-Flash-Next a native family, so the patched branch this
build used to require is gone. Three changes were made to meet the native
engine. No weights were requantized; the trunk, the MoE experts and the
draft head are the same oQ4e tensors as the MLX build.
| change | why |
|---|---|
| trunk RMSNorm gains shifted by +1 | oMLX writes zero-centered gains and applies normed*(1+w); MTPLX packs carry absolute gains and apply normed*w. 2.10.0 adds the +1 only for checkpoints its raw-HF discriminator recognises, and this pack reads as already-converted. Without the shift the model loads, serves, and benchmarks normally while emitting pure token soup. 155 tensors across 8 norm families. |
n-gram table repackaged into ngram-table.safetensors |
2.10.0 gathers the 320,001,536-row table row-wise from SSD and builds no weight parameter for it. Our build had it inline as 128 ngram_embedding.shards.N tensors, which the native loader rejects as parameters not in the model. Rows copied byte-for-byte; quantization (4-bit, group 32) unchanged. |
config.json quantization keys re-prefixed |
oQ writes model.embed_tokens; the loader tests language_model.model.embed_tokens. Without the prefix every per-module override was missed and the 8-bit embedding was built at 4-bit. 609 of 625 keys remapped. |
config.json also now declares ngram_sidecar: true. Shards 2 – 6 held nothing
but n-gram rows and are empty in this revision.
The norm shift is the one worth dwelling on: it is invisible to every health
check. The server reports ready, finish_reason is normal, token counts are
normal, and throughput is normal. Only reading the output reveals it.
Quick start## Quick start## Quick start
Apple Silicon. Resident n-gram wants ~108 GiB of unified memory; streaming the
table from SSD instead needs ~78 GiB plus a hot cache.
pip install mtplx
hf download grant-ai/Qwen3.8-Flash-Next-Abliterated-MTPLX-4bit \
--local-dir ~/models/qwen38-abliterated-mtplx
MTPLX_NGRAM_RESIDENT=1 mtplx serve \
--model ~/models/qwen38-abliterated-mtplx \
--model-id qwen38-abliterated \
--generation-mode mtp --depth 1 \
--port 8000
On a Mac with less than ~160 GB of RAM, drop MTPLX_NGRAM_RESIDENT=1 — MTPLX
streams the table from SSD automatically below that threshold and the model
still runs, about 12% slower.
Then use the OpenAI-compatible endpoint:
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model": "qwen38-abliterated",
"messages": [{"role": "user", "content": "Explain MoE routing."}],
"max_tokens": 1024, "temperature": 1.0, "top_p": 0.95, "top_k": 20}'
Add --generation-mode mtp --depth 2 to enable speculative decoding, but see Speculative decoding status first: it is currently slower than ar on this build.
Settings that matter
| setting | value |
|---|---|
| sampler | temperature 1.0, top_p 0.95, top_k 20 (from the model's own generation_config.json) |
| generation mode | --generation-mode mtp --depth 1 — 1.48x over ar |
| n-gram residency | MTPLX_NGRAM_RESIDENT=1, worth ~14%. Auto-arms at >=160 GB RAM |
| MTP depth | 1 is fastest (50.1), 2 is close (47.1), 3 falls back to ~42. Runtime caps at 3 |
--generation-mode auto |
measured 47.4 — within noise of fixed depth 1 but with wider spread. Fixed --depth 1 is the better default |
| n-gram hot cache | MTPLX_NGRAM_HOT_MB (default 1024) when streaming. Raising it to 8 GB or 24 GB measured within noise |
| thinking | chat_template_kwargs: {"enable_thinking": false} to disable |
| reasoning effort | --reasoning-effort low | medium | xhigh, or reasoning_effort in the API |
Reasoning effort defaults to xhigh for chat, the most expensive setting,
and is the largest lever on time-to-answer: at xhigh roughly half the
generated tokens are reasoning rather than answer. 2.10.0 added a real--reasoning-effort flag (the family used to ignore the field) and defaults
coding-agent lanes to medium.
Speculative decoding
Speculation works and is worth taking. Measured on MTPLX 2.10.0 stock, M3 Ultra,
1024-token cap, median reported (8 reps for the recommended row, 4 for the rest):
| mode | tok/s | vs AR | spread |
|---|---|---|---|
| MTP depth 1, resident n-gram | 50.1 | 1.48x | 8.7% |
| MTP depth 2, resident n-gram | 47.1 | 1.39x | 14.2% |
| AR, resident n-gram | 33.9 | 1.00x | 1.1% |
| AR, streamed n-gram | 30.0 | 0.88x | 4.2% |
MTP throughput carries more run-to-run variance than plain decoding because
acceptance depends on what is being generated; the AR rows are steady to within
1 – 2%, the MTP rows move by 6 – 9%. Longer generations narrow it.
Rewrite-shaped work is much faster still. MTPLX 2.10.0's batched
context-copy lane proposes up to 24 tokens as one block when the output repeats
an n-gram from the prompt, verified with the same acceptance math as MTP. On an
edit task — hand the model a module and ask for the whole thing back with one
change — decode measured 64.6 tok/s median against 46.1 with the lane
disabled (MTPLX_CONTEXT_COPY_BATCHED=0), a 40% gain, with one run reaching
113.9 tok/s. The spread is wide by nature: the win scales with how much of the
answer already exists in the prompt. The lane is on by default, so coding-agent
workloads get this without configuration. All outputs were checked for
correctness, not just speed.
Long context holds up. At a 17.5k-token prompt, decode measures 38.8 tok/s
(median of 3, spread 8.9%) — 80% of the short-prompt rate — with 2.10.0's QSA
rows-gather lane active. Prefill on that prompt runs about 900 tok/s. Answers
were checked for correctness, not just fluency: asked to count the sections in a
17.5k-token document, the model answered 236 against a ground truth of 236.
This reverses what earlier revisions of this card said. Through the 2.9.x
patched branch, speculation measured slower than plain decoding and the card
recommended ar. That was the RMSNorm defect: with zero-centered gains left
unshifted the trunk emits noise, so the draft head had nothing coherent to
predict against. The throughput numbers looked plausible the whole time — only
reading the generated text exposed it. With the gains shifted, acceptance
translates into the speedup the draft head always deserved.
Differences from the MLX build
The MLX build
is the same weights served by oMLX. Pick between them on these points:
| MLX (oMLX) | MTPLX (this build) | |
|---|---|---|
| runtime | stock oMLX | stock MTPLX 2.10.0 |
| vision / image input | yes | no |
| MTP speedup | ~1.95x | 1.48x |
| MTP acceptance | 71.3% | 96.5% |
| n-gram SSD offload | yes, ~76 GiB resident | yes, 29.8 GB sidecar |
Vision is text-only under MTPLX. As of 2.10.0 the qwen4_exp backend does
not construct the vision tower at all — upstream calls text serving the
"phase-1 contract" and reports image input as not yet supported for this
family. The vision weights are still present and indexed in this pack (333
tensors), so it is ready the moment MTPLX ships the tower.
⚠️ Do not send image requests to this build. In
--generation-mode arthe
server refuses them cleanly (image content requires MTP generation mode),
but inmtpmode it accepts them and returns token soup rather than
erroring, because the tower is never initialised. Use the
MLX build
under oMLX for image input; it serves the same weights with vision working.
Speculative decoding is slower here than under oMLX, despite higher draft
acceptance. That is not the draft head: substituting a full BF16 head for the
4-bit one changed throughput by 1.6%. It appears to be the cost of the generic
snapshot and restore path that qwen4_exp's four-slot recurrent cache requires.
If you want maximum tokens per second, use the MLX build.
Personas
Ships with the stock Qwen chat template as the default, which injects no
system prompt. The abliterated weights carry the behaviour on their own.
The upstream Blackfrost "Qwentium" persona is included as an opt-in: a ~1,200
token compliance-oriented system prompt prepended to every request. Switch with
the bundled script, then restart your server:
bash switch_persona.sh blackfrost # persona on
bash switch_persona.sh qwen # back to stock (default)
bash switch_persona.sh status # which is active
The template lives in two places, chat_template.jinja and the chat_template
key inside tokenizer_config.json, and different runtimes read different ones,
so the script switches both. It is safe against an hf download cache checkout:
it replaces symlinks rather than writing through them.
Measured on the MLX build, the persona costs about 0.8 s of prompt-processing
latency per request and leaves decode throughput unchanged.
Technical details
| architecture | qwen4_exp, hybrid Gated-DeltaNet + sparse attention, MoE, PLE n-gram embeddings |
| quantization | oQ4e mixed precision, affine, group size 32 base |
| bit widths | 4 / 5 / 6 / 8 bit, group sizes 32 / 64 / 128 |
| MTP head | 4-bit, iMatrix-calibrated, shipped as mtp.safetensors |
| vision | present in files, unused by this runtime |
| runtime | MTPLX 2.10.0 (stock) |
| n-gram table | 320,001,536 rows, 4-bit g32, shipped as ngram-table.safetensors (32.0 GB) |
How it was made
- Started from
Qwen3.8-Flash-Next-Abliterated-MLX-4bit,
itself oQ4e-quantized from the abliterated BF16 weights with iMatrix
calibration. - Packaged through
mtplx forge buildwithbody_bits: 0, so the body was
not requantized; the shards pass through unchanged. - The MTP head was extracted to a standalone
mtp.safetensorssidecar, still
4-bit and still iMatrix-calibrated. config.jsonquantization entries were remapped to the module paths the
MTPLX loader tests, and entries were added for the projection pairs MTPLX
fuses at load time.- For MTPLX 2.10.0: the inline n-gram shards were repackaged byte-for-byte into
a singlengram-table.safetensors, the quantization keys were re-prefixed
withlanguage_model., and the trunk RMSNorm gains were shifted by +1 to the
absolute convention MTPLX packs use. No weights were requantized at any
step.
The one measurement worth repeating: the 4-bit iMatrix draft head matched a full
BF16 head on prediction quality (96.5% / 94.2% against 95.4% / 91.9% at depth 2),
so quantizing the draft head cost nothing.
Limitations
- No vision. Text only under this runtime, and image requests in
mtpmode return garbage instead of an error. - Requires MTPLX 2.10.0 or newer. Earlier releases cannot load it.
- Capability benchmarks have not been run; only conversion correctness and
throughput were verified. - Refusal behaviour was not measured on this build. "Abliterated" describes the
upstream weights, not a property this conversion tested. - Speculative decoding reaches 1.48x here against ~1.95x for the same weights under oMLX.
- Speed depends heavily on machine, context length, reasoning effort, and
thinking mode.
Credits
- Qwen for Qwen3.8-Flash-Next.
- Blackfrost-AI for the abliterated BF16
weights this build descends from. - @maceip for
PR #380, theqwen4_exp
support this build runs on. - MTPLX for the runtime, and for making
Qwen3.8-Flash-Next a native family in 2.10.0. - oMLX for the oQ mixed-precision quantizer used to produce the weights.
Disclaimer and licence
This is an independent, unaffiliated conversion. It is not endorsed by,
developed with, developed by, sponsored by, or associated with Qwen, Alibaba
Cloud, Blackfrost-AI, MTPLX, or oMLX. None of them reviewed, approved, or
contributed to this repository, and none of them bear any responsibility for it.
It does not represent Qwen or Qwen3.8-Flash-Next. The behavior of this model
is not the behavior of the official Qwen release. Refusal behavior has been
removed by a third-party modification, and the source checkpoint's injected
system prompt has been made optional here rather than default. Do not treat any
output of this model as representative of, or attributable to, the upstream
model or any party named in this card. "Qwen" is used solely to identify the
base architecture.
Changes made to the upstream work: quantized to mixed-precision oQ4e with
iMatrix calibration, body and MTP draft head together; the MTP head repackaged
as a standalone sidecar; config.json quantization keys remapped and extended
for the MTPLX loader; the source checkpoint's chat_template.jinja and thechat_template key in tokenizer_config.json replaced with the stock Qwen3.8
template, with the source checkpoint's injected system prompt retained in the
repository as an optional swap rather than a default. No weights were modified
beyond quantization; the abliteration itself is upstream work.
Licence: Qwen Community License 1.0, inherited from the upstream model and
weights. Your use is also subject to the licence terms of those upstream works.