license: apache-2.0
language:
- en
- zh
base_model: - huihui-ai/Huihui-Qwen3.8-27B-abliterated
- unsloth/Qwen3.8-27B-NVFP4
tags: - paiton
- w3a4
- w3rot
- 3-bit
- abliterated
- rdna4
Qwen3.8-27B abliterated — Paiton W3Rot INT3 (W3A4) overlay
A 3-bit overlay for the Paiton paiton-w3rot-v1 format, built from the
huihui-ai abliterated checkpoint
and calibrated with GPTQ (block-diagonal Hessian, 65,536 tokens).
It is an overlay, not a full model: only the attention / GDN / MLP projections are here (64 shards,
1,296 tensors). embed_tokens, lm_head, norms, A_log, conv1d and dt_bias still come from the
NVFP4 base (unsloth/Qwen3.8-27B-NVFP4, revision f0b7c9e).
Usage
Mount this directory at /models/w3rot-abl-gptq, mount the NVFP4 base at /models/target, and setPAITON_W3_A4_DIR=/models/w3rot-abl-gptq in the Paiton container. The runtime merges the overlay over the base;
no other change is needed.
Results
Same harness, one Radeon AI PRO R9700 (32 GB), 262K context profile, thinking off:
| Benchmark | This overlay | Official W3A4 |
|---|---|---|
| MMLU-Pro, 14×200 random, 0-shot | 57.86% | 57.82% |
| GSM8K 5-shot (1,319) | 93.10% | 95.30% |
| HumanEval pass@1 (164) | 90.24% | 93.90% |
| Needle @ 61,440 tokens (160) | 160/160 | 100 |
Single-stream decode (BetterBench 0.6.1, 20 passes, weighted): 155.0 tok/s vs 174.3 for the official 262K mode
on the same card; 8-way concurrency 436.8 vs 458.9. The gap is host-side (virtualized guest on an older Xeon),
not the weights: the official weights measured in the same container run at the same speed.
Build
Scripts and notes: https://github.com/halonc/paiton-vllm-plugin-27b-w3a4-abliterated
Licence and caveats
Apache-2.0. Derived from Apache-2.0 checkpoints; the upstream Paiton plugin is Apache-2.0.
Abliteration removes a refusal direction, not policy; outputs are the user's responsibility.
Research and personal use only.