license: apache-2.0
base_model: windowsxp811203/Qwen3.8-27B-Abliterated
pipeline_tag: text-generation
library_name: mlx
tags:
- mlx
- mlx-vlm
- qwen3_5_mtp
- mtp
- speculative-decoding
- draft-model
- qwen3.8
- qwen
- abliterated
language: - en
- zh
Qwen3.8-27B-Abliterated-MTP-bf16
The MTP drafter weights split out of
windowsxp811203/Qwen3.8-27B-Abliterated, in bf16,
for mlx-vlm speculative decoding.
This is not a standalone model. Pair it with a Qwen3.8-27B-Abliterated MLX target; the target supplies the token
embeddings and the LM head at runtime. Use the abliterated drafter with abliterated targets only: the parent's
abliteration edited the MTP head together with the trunk (ABLIT_META.edit_mtp: true), so the base-Qwen drafters
(mlx-community/Qwen3.8-27B-MTP-*) are mismatched with these targets and vice versa.
Use with mlx-vlm
pip install -U mlx-vlm
hf download windowsxp811203/Qwen3.8-27B-Abliterated-MLX-MTP --include "oQ4e/*" --include "drafter/*" --local-dir ./mlx-mtp
python -m mlx_vlm.generate --model ./mlx-mtp/oQ4e --draft-model ./mlx-mtp/drafter \
--prompt "Write a quicksort in Python." --max-tokens 512
--draft-kind mtp is detected automatically from model_type. Measured on an M5 Max 128 GB with mlx-vlm
0.6.17, temperature 0: single prompt (36 tokens), 300 generated tokens, best run per arm across 3 sessions: oQ4e 31.1 → 39.9 tok/s (1.29×, 90.6% of drafts accepted; plain runs 24.0–31.1 n=5, drafter runs 33.1–39.9 n=3); oQ8e 18.0 → 34.8 tok/s (1.94×, 88.4% of drafts accepted; plain runs 14.5–18.0 n=5, drafter runs 26.4–34.8 n=3); oQ6e 22.1 → 20.2 tok/s (0.91×, 88.4% of drafts accepted; plain runs 18.9–22.1 n=5, drafter runs 16.9–20.2 n=3). The 6-bit target gains nothing from this path in our runs (the 6-bit affine kernels appear to leave no headroom for the multi-token verify pass; oQ6e+drafter is even slower than oQ8e+drafter) — use oMLX's in-checkpoint MTP there.
Targets (folders of the project repo): oQ4e/ · oQ6e/ ·
oQ8e/ · bf16/ (the drafter path was measured on the three oQ targets only). Those checkpoints also carry the
MTP head inside the file for oMLX; this repo is only needed for the mlx-vlm--draft-model path.
Model details
- Model type:
qwen3_5_mtp· block size 3 · 15 tensors · bf16 · 810 MiB (849,400,335 bytes) - Extracted with
mlx_vlm.split_mtp(0.6.17) directly from the HF bf16 parent; norms carry MLX's +1 RMSNorm convention. - Speculative decoding is distribution-preserving: drafts are verified against the target's own distribution, so acceptance only changes speed. It is not bit-exact under bf16 kernels — greedy outputs can diverge after a near-tie token.
Support / 打賞
If these models are useful to you, tips are appreciated — they pay for the GPU time.
如果這些模型對你有幫助，歡迎打賞，用於支應算力成本。
USDT (TRC20) · TPTo32r7vKazpTNaFqfFZ2ztoK1DG88888
Disclaimer
Inherits the Apache-2.0 license of the base model. The parent model will not refuse; you are responsible for your use.