license: apache-2.0
base_model: huihui-ai/Huihui-Qwen3.6-27B-abliterated
tags:
- nvfp4
- mxfp8
- dflash
- speculative-decoding
- abliterated
- vllm
- qwen3.6
Huihui-Qwen3.6-27B-abliterated · NVFP4 + DFlash MXFP8 draft · 128k on 2×16GB
Qwen3.6-27B (abliterated), NVFP4-quantized, with an MXFP8-quantized DFlash speculative-decoding draft — running 128k context at ~128 tok/s average (up to 183 tok/s) on two RTX 5070 Ti 16GB cards, one of them on an M.2 riser.
To our knowledge this is the first vLLM-loadable (modelopt format) quantized DFlash draft for Qwen3.6-27B on HF (searched 2026-08-13: existing DFlash quants are GGUF / MLX / OpenVINO only, including the abliterated+DFlash combos).
TL;DR
| Hardware | 2× RTX 5070 Ti 16GB, TP2 — GPU1 sits on an M.2 → PCIe 4.0 x4 riser (¼ bandwidth, NCCL P2P disabled) |
| Context | 131,072 tokens (KV pool 131,378, FP8 KV) |
| Single-stream decode | 163–183 tok/s — median 163–168 on short code prompts (10-run), 183.5 sustained on Spec-Bench math_reasoning (n=25, 20 warmups) — single-RTX-5090 territory from a cheap asymmetric dual-card box |
| Spec-Bench avg | 128.2 tok/s across 8 categories, +31% vs the same model with its native MTP head (K=3) |
| Acceptance length | 3.6–5.7 @ K=8 (per-position decay is a gentle slope, not a cliff) |
What's in the box
├── model-0000{1..3}-of-00003.safetensors # 27B backbone, NVFP4 (FP8 attn/KV), MTP tensors physically removed
├── config.json / hf_quant_config.json # modelopt mixed quant config
└── draft/
├── model.safetensors # DFlash draft, MXFP8 (self_attn excluded), 1.9 GB
└── ... # upstream z-lab card kept for provenance
Provenance
| Piece | Source | What we did |
|---|---|---|
| Backbone | huihui-ai/Huihui-Qwen3.6-27B-abliterated (orthogonal ablation / uncensored) | Self-quantized to NVFP4 with NVIDIA TensorRT Model Optimizer, following NVIDIA's own recipe for Qwen3.6-27B (nvfp4_fp8_mha: W4A16 NVFP4 weights, FP8 attention, FP8-cast KV). Verified layer-by-layer identical structure vs NVIDIA's official quant. Then physically stripped the 15 native mtp.* tensors (this bundle uses DFlash instead — vLLM would otherwise pick the MTP weights up from the shards). |
| Draft | z-lab/Qwen3.6-27B-DFlash (BF16, 3.2 GB) | Self-quantized to MXFP8 (group 32, *self_attn* excluded): 3.22 → 1.90 GB. Acceptance is unharmed (pos0 ≈ 80–90%, see below). MXFP8 over NVFP4 on purpose — draft precision buys acceptance; we didn't need the extra VRAM. |
| Engine | vllm/vllm-openai:nightly |
Pinned digest below — nightly moves, the digest doesn't. |
vllm/vllm-openai@sha256:a671d5fcda70fe9ac6f245f9780821de459fb4ee22c018fd07a0f10a55279bf9
Run it (the exact 128k command)
docker run -d --name hh27dflash --gpus all \
-e NCCL_P2P_DISABLE=1 \
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
-v /path/to/this-repo:/model \
-v /path/to/this-repo/draft:/draft \
-p 8002:8000 \
vllm/vllm-openai@sha256:a671d5fcda70fe9ac6f245f9780821de459fb4ee22c018fd07a0f10a55279bf9 \
/model --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 \
--trust-remote-code --served-model-name hh27 \
--kv-cache-dtype fp8 --attention-backend TRITON_ATTN \
--mamba-cache-dtype bfloat16 --mamba-ssm-cache-dtype bfloat16 \
--gpu-memory-utilization 0.96 --kv-cache-memory 2700000000 \
--language-model-only --max-model-len 131072 \
--max-num-seqs 3 --max-num-batched-tokens 1024 --enable-chunked-prefill \
--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8,"attention_backend":"TRITON_ATTN","max_model_len":32768}' \
--default-chat-template-kwargs '{"enable_thinking": false}' \
--reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice
The five flags that make 128k possible on 2×16GB (each one paid for in blood)
This model's KV cost is brutal: 16 full-attention layers × head_dim 256 = 16,384 B/token/GPU — twice a normal 27B. 128k does not fit naively.
--language-model-only— skips loading the vision tower, frees ~0.5 GiB/GPU. Language weights are bit-identical; this is the flag that closes the 128k gap for free.--mamba-cache-dtype bfloat16+--mamba-ssm-cache-dtype bfloat16— default fp32 mamba state is pure waste; bf16 grows the KV pool by ~22% with byte-identical outputs in our tests.--kv-cache-memory 2700000000— 2.85e9 boots fine and then OOMs the engine on the first heavy prefill (34 MB activation ask vs 18 MB free). Leave ~150–250 MB headroom; the pool is exactly 131,378 tokens at 2.7e9.--attention-backend TRITON_ATTNand"attention_backend":"TRITON_ATTN"inside--speculative-config— the drafter does not inherit the CLI flag, and sm_120 FLASH_ATTN doesn't support FP8 KV. Miss either one and it dies."max_model_len":32768inside--speculative-config— the draft's context can be set independently; leave it unset and it inflates to 128k, eating KV for nothing.
Also load-bearing: --max-num-seqs 3 (4 OOMs, 8 hits NCCL errors at this watermark), --max-num-batched-tokens 1024 (halves prefill activation spikes), K=8 (K=15 measured slower — deep draft tails fall under the ~12% break-even acceptance).
Benchmarks
All numbers: this exact config, vllm bench serve, Spec-Bench protocol, --ignore-eos, 20 warmups, concurrency 1, thinking off. Same machine, same day, same question sets as the MTP baseline.
DFlash (this repo, K=8) vs native MTP head (K=3)
The same abliterated NVFP4 backbone can run its native single-layer MTP head instead (K=3, FLASHINFER backend, 128k) — full baseline config at the bottom of this section. Head-to-head:
| Category | n | MTP tok/s | DFlash tok/s | Δ | accept_len MTP → DFlash |
|---|---|---|---|---|---|
| math_reasoning | 25 | 117.6 | 183.5 | +56% | 3.44 → 5.69 |
| coding (Spec-Bench) | 25 | 117.8 | 167.4 | +42% | 3.25 → 4.61 |
| translation | 25 | 116.1 | 151.6 | +31% | 3.21 → 4.28 |
| qa | 25 | 109.9 | 134.2 | +22% | 2.96 → 3.57 |
| rag | 25 | 89.6 | 111.9 | +25% | 3.30 → 4.51 |
| summarization | 25 | 83.9 | 100.1 | +19% | 3.01 → 3.67 |
| agent (held-out) | 25 | 77.9 | 95.5 | +23% | 3.33 → 4.60 |
| code (held-out, long prompts) | 157 | 69.6 | 81.3 | +17% | 3.10 → 4.17 |
| mean (8 categories) | 332 | 97.8 | 128.2 | +31.1% |
332 questions per side (664 fixed-length generations total), all with 20 warmup requests per category — not a handful of cherry-picked prompts.
Median ITL is ~25–26 ms for both — per-step cost is identical, the entire win is tokens-per-step.
Note on acceptance rate: don't compare it across different K — the denominator is K, so deeper drafts dilute the average by construction. Compare acceptance length and tok/s.
Per-position acceptance (DFlash, K=8, coding)
pos: 0 1 2 3 4 5 6 7
acc: 88.7 75.5 57.6 48.1 38.7 31.1 27.4 18.9 (%)
A gentle slope — unlike recursive MTP heads, which cliff after position 2. That flat tail is why DFlash affords K=8 while MTP saturates at K=3.
Rough single-stream feel (10 runs, short code prompt, 512 tok, warmed)
min 149.8 | median 163.1 | mean 161.2 | max 179.5 tok/s
On a box where one GPU talks through an M.2 riser at PCIe 4.0 x4. This is single-5090-class decode from two lame 16GB cards.
MTP baseline config (for reproducibility)
The MTP numbers above were produced with: native single-layer MTP head, {"method":"mtp","num_speculative_tokens":3}, FLASHINFER backend (VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE=134217728), FP8 KV, --kv-cache-memory 3000000000, --language-model-only, 131,072 context, seqs 8, batched 2048, same nightly image. Full script in the companion repo.
Credits
- huihui-ai — the abliterated base
- z-lab — DFlash (project page), the draft this quant derives from
- NVIDIA TensorRT Model Optimizer — quant recipe
- vLLM — engine
Uncensored model: you are responsible for what you generate with it.