license: other
license_name: qwen-community-1.0
license_link: LICENSE
library_name: mlx
pipeline_tag: image-text-to-text
base_model: orcarouter/Qwen3.8-Flash-Next-Uncensored
base_model_relation: quantized
tags:
- mlx
- mlx-serve
- qwen4_exp
- apple-silicon
- mixed-precision
- 3-bit
- 4-bit
- 8-bit
- uncensored
- mtp
Qwen3.8-Flash-Next-Uncensored — MLX-Serve mixed 3/8-bit
An experimental MLX-Serve mixed-precision pack derived from
orcarouter/Qwen3.8-Flash-Next-Uncensored.
Routed MoE expert weights use 3-bit affine quantization (group size 64); the
remaining eligible projections use 8-bit quantization. The token embeddings
and the separate ngram_table.bin remain 4-bit. The MTP head and vision files
are included.
This pack is for the patched mlx-serve runtime used in the validation. It is
not a generic Transformers, mlx-lm, or mlx-vlm checkpoint.
The matching V2 runtime patch and validation manifests are inengine/README.md.
Which variant should I use?
Two MLX-Serve-specific mixed-precision variants are available:
| Variant | Positioning | Routed MoE experts | Other eligible projections | Token embeddings | N-gram table | Approx. pack size |
|---|---|---|---|---|---|---|
| 4/8-bit | Recommended default / quality-oriented | 4-bit | 8-bit | 4-bit | 4-bit | ~107 GB |
| 3/8-bit | Lower-memory experimental variant | 3-bit | 8-bit | 4-bit | 4-bit | ~86 GB |
Choose the 4/8-bit variant as the default starting point when memory
capacity is sufficient. It uses less aggressive quantization for the routed
MoE expert weights.
Choose the 3/8-bit variant when reducing model memory or storage
requirements is more important. Only the routed MoE expert weights are
reduced to 3-bit; the token embeddings and N-gram table remain 4-bit, and the
other eligible projections remain 8-bit.
These labels describe the intended trade-off between the two packs. They are
not a claim that the 4/8-bit variant is universally more accurate, or that
the 3/8-bit variant has universally equivalent quality or speed. Measured
quality and performance are workload-, context-, runtime-, and
hardware-dependent.
Runtime compatibility
These are specialized checkpoints for the patched
mlx-serveruntime used
by this repository. They are not generic Transformers,mlx-lm, ormlx-vlmcheckpoints. Do not rely on Hugging Face's automatically
generatedmlx-lm/mlx-vlmusage examples for these packs; follow this
repository'sengine/README.mdand runtime instructions instead.
Validation notes
Results reported for each variant should be interpreted only under the
conditions documented on that variant's model card. Scores obtained with
different task sets, runtime revisions, generation limits, or evaluation
procedures should not be compared directly.
The 3/8-bit variant has additionally been validated on a Mac Studio M3 Ultra
with 96 GB unified memory using its matching patched MLX-Serve V2 runtime. In
that workload, it reduced measured MLX peak memory relative to the tested
4/8-bit reference. This is a workload-specific observation, not a general
memory or quality guarantee.
Download
hf download yomie4343/Qwen3.8-Flash-Next-Uncensored-MLX-Serve-mixed-3-8bit \
--local-dir ./qwen-flash-next-mixed-3-8bit
Requirements
Use an Apple Silicon Mac with the matching patched mlx-serve V2 runtime and
keep the complete pack on a fast local SSD. The model files alone are not a
drop-in installation of the runtime patch.
Runtime
Use the V2 engine with the 3-bit kernel and these environment variables:
export MLX_SERVE_Q3_PACK=8
export MLX_SERVE_Q3_BYTES=1
export MLX_SERVE_NGRAM_WARM=0
export MLX_SERVE_CACHE_LIMIT=2147483648
./zig-out/bin/mlx-serve \
--model /absolute/path/to/Qwen3.8-Flash-Next-Uncensored-MLX-Serve-mixed-3-8bit \
--serve --host 127.0.0.1 --port 11234 --ctx-size 204800 \
--kv-quant 8 --prefill-chunk 1024 --no-pld
The model pack is about 86 GB on disk, including the approximately 32 GB
4-bit n-gram table. Keep the complete directory on a fast local SSD.
Validation on Mac Studio M3 Ultra, 96 GB, 2026-09-18 JST
The patched engine reached practical speed parity with the existing 4/8-bit
pack: across approximately 3K, 12K, and 47K input tokens, median prefill and
decode differences stayed below 0.4% in the repeated comparison. This is a
workload-bounded comparison, not a claim of universal statistical equivalence.
Actual 127,744–200,094-token inputs passed the fixed JSON and tool-retrieval
checks in all 6/6 runs for this pack. Its cumulative MLX peak in that phase
was 63.774 GB, about 14.7 GB below the 4/8-bit reference.
The original 25-task Hermes/TypeScript comparison was independently graded at
72/75 (three repeated runs of the same 25 tasks). The three failures were the
same binary-search-tree implementation type error (missing return values).
A separate fixed-8192-output supplement for the two tasks affected by Hermes'
adaptive output cap passed 6/6 across this pack; that supplement is reported
separately and is not pooled with the 72/75 score.
Limitations
This is an experimental quantization and runtime combination. The validation
sets are fixed and do not guarantee general coding quality, reasoning quality,
or performance on other hardware. The source model is uncensored and can
produce unsafe, incorrect, or offensive output. Use least-privilege tools and
application-level safeguards.
The Qwen Community License 1.0 is retained; see LICENSE and review
its terms before redistribution or commercial use.