license: other
license_name: qwen-community-license-1.0
license_link: LICENSE
base_model: orcarouter/Qwen3.8-Flash-Next-Uncensored
base_model_relation: quantized
pipeline_tag: image-text-to-text
tags:
- sushi
- mlx
- exl3
- 3bit
- abliterated
- qwen
- moe
Qwen3.8-Flash-Next-Abliterated-Sushi-3bpw
Experimental Sushi-format quantization made directly from the BF16 weights of
orcarouter/Qwen3.8-Flash-Next-Uncensored,
revision e096800036ec20da7e2442dcd4044a004d4e99fa. The 3-bit sibling of
Qwen3.8-Flash-Next-Abliterated-Sushi-2bpw.
The abliteration comes from that upstream checkpoint; this conversion adds no
further abliteration or fine-tuning. Refusal behavior has not been independently
characterized here.
Format and conversion
- Routed experts: uniform EXL3 K3, MCG codebook, window15, including MTP experts,
calibrated from BF16 (64 rows x 1024 tokens). - Everything else is byte-identical to the 2bpw pack (same sha256): affine8 dense
trunk, vision tower, MTP trunk, tokenizer, and the affine4/group32 n-gram sidecarngram_table.bin. Only the expert shards andconfig.json(expert_quant.k) differ. - Converter based on ExLlamaV3
151539c77abc7ab7425d30da7a4e8e3c5c154e7b, with an
isolated MCG-window15 encoder adaptation; its K2 and K3 output was checked
bit-exact against Sushi v1.2.1's own EXL3 decoder before conversion.
This is a Sushi pack, not a general ExLlamaV3 or Transformers checkpoint. 3bpw
refers to the routed-expert code rate. No lossless/BF16-equivalent quality claim
is made. The official Sushi-3bpw uses a private converter at window14; this pack
does not reproduce that recipe, and no KLD measurement has been made for it.
Running
The complete download is about 79.22 GiB (including the 29.8 GiB disk-backed
n-gram sidecar). If you already have the 2bpw pack, only the expert shards differ.
hf download sanasol2008/Qwen3.8-Flash-Next-Abliterated-Sushi-3bpw \
--local-dir ~/.sushi/models/Qwen3.8-Flash-Next-Abliterated-Sushi-3bpw
Expert weights are about 1.5x the 2bpw pack. On a 48 GiB Mac they are streamed
from SSD (--ssd-budget-gb), so decode is slower than 2bpw there; 64 GiB or more
keeps much more of the pack resident. The image check below passed on the public
Sushi v1.2.1 engine with --vision under a 22 GiB streaming budget; the patched
engine from the 2bpw card was not needed for it.
sushi serve --model ~/.sushi/models/Qwen3.8-Flash-Next-Abliterated-Sushi-3bpw \
--host 127.0.0.1 --port 12345 --ssd-budget-gb 22 --no-mtp --vision \
--kv-quant 8 --ctx-size 262144 --prefill-chunk 2048 --metrics \
--max-tokens 8192 --prefix-cache-entries 2 --prefix-cache-mem 1GB \
--prefix-cache-disk 10GB --temp 1
The 256k context is a configured ceiling, not a measured full-context guarantee.
Converter-reported calibration error on decoder layer 0, the one layer measured in
both runs: relative Frobenius error 0.029 / SQNR 43.4 dB at K3 versus 0.064 / 38.0 dB
for the 2bpw pack.
Validation
Tested on Apple M3 Max / 48 GiB unified memory with the public Sushi v1.2.1 engine
(--ssd-budget-gb 22 --no-mtp --kv-quant 8).
Functional checks (thinking off, temperature 0): arithmetic, a Russian instruction,
an image OCR check that transcribed all three fields including numbers, and a tool call
returning get_weather with city=Belgrade all passed. These establish basic
functionality, not model-wide quality, BF16 equivalence or a measured refusal rate.
Controlled decode comparison: a fresh server per pack, 16k context, prefill chunk 512,
prefix cache off, the same 40-token prompt, 256 generated tokens, nothing else running.
| Pack | Cold decode | Warm decode, mean of runs 2–3 |
|---|---|---|
| Abliterated 2bpw | 18.36 tok/s | 28.91 tok/s |
| This 3bpw pack | 11.88 tok/s | 12.88 tok/s |
Short-prompt expert-cache measurements on a 48 GiB machine that cannot hold the
3bpw experts resident; they are not long-context agent throughput, and a Mac with
more memory will see a different ratio. MTP, video and long-context behavior were not validated.
License and attribution
The upstream LICENSE is included verbatim: Qwen Community License 1.0. The upstream
model card labels Apache 2.0, but its actual bundled license differs; this repository
preserves the bundled license rather than asserting a new Apache license.
Credit to Qwen for the base model, OrcaRouter for the BF16 derivative,
turboderp-org for ExLlamaV3, and beamivalice for Sushi.