license: apache-2.0
base_model: huihui-ai/Huihui-Qwen3.8-27B-abliterated
library_name: vllm
pipeline_tag: image-text-to-text
tags:
- qwen3_5
- qwen3.8
- abliterated
- uncensored
- compressed-tensors
- w8a16
- int8
- quantized
- vllm
- dflash2
- speculative-decoding
- vision
- conversational
base_model_relation: quantized
Huihui-Qwen3.8-27B-abliterated · INT8 W8A16 · DFlash2
Fast, near-lossless Huihui abliterated Qwen3.8-27B for dual RTX 3090s.
Base model · Qwen3.8-27B · vLLM · llm-compressor
[!NOTE]
A numerical INT8 W8A16 quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated, an abliterated fine-tune of Qwen/Qwen3.8-27B, served with the DFlash2 drafter for speculative decoding. All model credit belongs to huihui-ai and Qwen; this repository changes numerics only. The detailed engineering notes live in TECHNICAL.md.
The short version
Huihui-Qwen3.8-27B-abliterated is a big model. This build makes it fast on consumer hardware:
- Up to ~117 t/s decode on two RTX 3090s at full 262K context (family baseline)
- ~1.4x to 1.5x faster than the same model with native MTP4 speculation (family baseline, pending own bench)
- ~2.2x faster than plain autoregressive decoding (family baseline, pending own bench)
- 98.5% top-1 agreement with the abliterated BF16 model (measured: mean KLD 0.000694)
- 28 GiB on disk (6 shards), 2.02 GiB drafter, full 262K context on 2×24 GB
The KLD and audit numbers above are this build's own measured evidence. Decode
throughput figures are the family baseline from the identical sibling build;
this build's own serve run is pending (see Status).
How fast
Measured on this build: KLD 0.000694 nats mean, 98.5% top-1. Decode numbers
below are the family baseline (single stream, same engine, same 262K context,
cold cache). Generation tokens per second.
| Prompt tokens | Autoregressive | MTP4 | DFlash2 | DFlash2 vs MTP4 |
|---|---|---|---|---|
| 128 | 47 | 77 | 117 | 1.5x |
| 2,048 | 47 | 73 | 103 | 1.4x |
| 8,192 | 47 | 75 | 102 | 1.4x |
DFlash2 proposes 7 draft tokens per step and the target verifies them all in
one pass. Native MTP proposes 4. More drafts per verification step means more
accepted tokens and less idle GPU time.
How close to the original
| BF16 original | This build | |
|---|---|---|
| Size | ~56 GiB | 28 GiB (6 shards) |
| Mean KLD | 0 | 0.000694 |
| Top-1 agreement | 100% | 98.5% |
KLD is a fancy way of asking "does the quantized model pick the same next
token as the original?" Lower is closer. A mean of 0.000694 nats means the
INT8 weights and the BF16 weights are nearly interchangeable. Halving the size
is what lets the full 262K context fit on two 3090s.
What you get
- Full 262,144-token context on 2×RTX 3090 (intended, pending serve)
- Vision tower, thinking controls, and tool calling intact, same as upstream huihui-ai/Huihui-Qwen3.8-27B-abliterated
- Native MTP removed: DFlash2 replaces it, so there is no dead weight in the repo
- Runs on vLLM with the club-3090 patch set (DFlash2 is a new spec-decoder, still in PR review upstream)
Run it
Three steps. The full command with every flag is in TECHNICAL.md.
# 1. Models (pulls into ~/.cache/huggingface)
hf download lued/Qwen3.8-27B-huihui-abliterated-INT8-W8A16-DFlash2
hf download lued/Qwen3.8-27B-DFlash2-W8
# 2. Patches (DFlash2 support is not in a released vLLM yet)
git clone https://github.com/noonghunna/club-3090.git
# 3. Serve (condensed; full command in TECHNICAL.md)
export CLUB3090="$HOME/club-3090"
podman run --rm --replace --device nvidia.com/gpu=all --ipc=host -p 8080:8080 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-v "$CLUB3090/models/qwen3.6-27b/vllm/patches/vllm-pr52816-dflash2":/etc/club3090/pr52816:ro \
-v "$CLUB3090/models/qwen3.6-27b/vllm/patches/vllm-pr48375-mamba-drop-eagle-block":/etc/club3090/pr48375:ro \
--entrypoint bash docker.io/vllm/vllm-openai:nightly-5a4c8d99242e9e069b604d0e9b969e77f7dd501d \
-c 'bash /etc/club3090/pr48375/install.sh || exit 1; bash /etc/club3090/pr52816/install.sh || exit 1; exec vllm serve "$@"' -- \
lued/Qwen3.8-27B-huihui-abliterated-INT8-W8A16-DFlash2 \
--tensor-parallel-size 2 --max-model-len 262144 --kv-cache-dtype fp8_e4m3 \
--speculative-config '{"method":"dflash","model":"lued/Qwen3.8-27B-DFlash2-W8","num_speculative_tokens":7}'
Status
Built, audited, and published 2026-08-20 to
lued/Qwen3.8-27B-huihui-abliterated-INT8-W8A16-DFlash2
at commit 26c9c9bc. STRUCTURAL AUDIT PASS (401 packed / 783 preserved,
independently re-verified tensor-by-tensor), DFLASH2 AUDIT PASS (15 MTP
stripped, embed packed), shared drafter audit PASS. KLD measured: mean
0.000694 nats, top-1 0.9850, sanity gate PASS. Registered in the HF cache and
verified to resolve offline. Serve-validation on this rig is pending (family
baseline decode from the identical sibling build). The drafter ships
separately as
lued/Qwen3.8-27B-DFlash2-W8.
All engineering detail, audit evidence, and measured tables are in
TECHNICAL.md.