license: apache-2.0
base_model: huihui-ai/Huihui-Qwen3.8-27B-abliterated
tags:
- int4
- w4a16
- awq
- gptq
- abliterated
- llm-compressor
- compressed-tensors
- vllm
library_name: transformers
Huihui-Qwen3.8-27B-abliterated-INT4-AWQ-GPTQ
Mixed-precision INT4 (W4A16) quantization ofhuihui-ai/Huihui-Qwen3.8-27B-abliterated,
built with llm-compressor.
25.1 GB. Calibrated on text generated by this abliterated model itself, not by
stock Qwen — see below, it matters.
Recipe
| component | precision |
|---|---|
mlp.{gate,up,down}_proj, layers 0–55 |
INT4, group-32, asymmetric (~4.6 effective bits) |
mlp.{gate,up,down}_proj, layers 56–63 |
INT8 W8A16 |
self_attn.{q,k,v,o}_proj |
INT8 W8A16 |
linear_attn.{in_proj_qkv,in_proj_z,out_proj} (GDN) |
INT8 W8A16 |
lm_head, embed_tokens, norms, GDN state params, vision tower |
BF16 |
AWQ per-input-channel scaling (folds into the norm weights — zero size cost), then
GPTQ Hessian error compensation. Runs on anything Turing or newer; no FP8/FP4 hardware needed.
Calibration: self-distilled from the abliterated model on a balanced Nemotron-v2 prompt
blend (25% code, 25% math, 20% STEM, 20% chat, 10% multilingual).
Benchmarks
Measured against the abliterated BF16 model as its own reference — not stock Qwen —
so the numbers reflect quantization damage only, not the effect of abliteration.
142,727 tokens plus 200 free greedy generations. vLLM 0.27.1, TP=2, 2×B300.
| build | size ↓ | top-1 ↑ | near-tie ↓ | moderate ↓ | confident ↓ | certain ↓ | divmed ↑ | tok/s ↑ |
|---|---|---|---|---|---|---|---|---|
| this model (INT4 AWQ+GPTQ) | 25.1 GB | 96.45% | 21.76% | 2.92% | 0.88% | 0.12% | 41 | 4551 |
| NVFP4 sibling (AWQ+AutoRound) | 24.7 GB | 92.98% | 34.02% | 9.83% | 1.80% | 0.20% | 27 | 10702 |
| earlier build (base-model calibration) | 24.7 GB | 91.79% | 37.58% | 11.21% | 3.54% | 0.21% | 20 | 10685 |
Sizes are on-disk tensor bytes and include the ~0.85 GB BF16 MTP head.
Columns. top-1 is raw argmax agreement with the BF16 abliterated model. The bucket
columns are disagreement rates split by how confident the reference was at that position
(top1−top2 logprob margin): near-tie <0.5, moderate 0.5–2, confident 2–5, certain >5.
Only confident and certain are real damage. divmed is the median token index at
which free greedy generation first diverges.
Perplexity is excluded — on this model family it is anti-correlated with quality.
Calibration matters more than abliteration
An earlier build of this model used the stock-Qwen calibration set and a weaker recipe, and
landed at 3.54% confident damage. Regenerating the calibration from the abliterated model
itself brings that to 0.88%.
That also answers a question worth stating plainly: abliterated weights are not intrinsically
harder to quantize. With matched recipe and self-distilled calibration this model reaches
0.88% confident damage, against 0.93% for the same recipe on stock
Qwen3.8-27B. The earlier gap was the calibration and recipe, not the abliteration.
INT4 vs NVFP4 on this model
The INT4 build is substantially more faithful than the NVFP4 one (confident 0.88% vs 1.80%,
paired McNemar z = 13.2; certain 0.12% vs 0.20%, z = 4.3), reproducing the same gap measured
on stock Qwen3.8-27B (0.93% vs 1.85%). At ~4.6 effective bits INT4 group-32 asymmetric gives
16 uniform levels plus a per-group zero point, against FP4's 8 non-uniform levels at 4.5 bits.
NVFP4's advantage is hardware: it decodes at 10702 tok/s here versus 4551. Pick the
NVFP4 sibling if you
are throughput-bound on Blackwell; pick this for fidelity, or on Ampere/Ada.
Usage
from vllm import LLM
llm = LLM("TelperionAI/Huihui-Qwen3.8-27B-abliterated-INT4-AWQ-GPTQ", tensor_parallel_size=2)
Speculative decoding (MTP)
The MTP head is included, in BF16, grafted from the abliterated base (not stock Qwen):
llm = LLM("TelperionAI/Huihui-Qwen3.8-27B-abliterated-INT4-AWQ-GPTQ", tensor_parallel_size=2,
speculative_config={"method": "mtp", "num_speculative_tokens": 2})
Qwen3_5ForConditionalGeneration does not carry mtp.* in its state dict, so
llm-compressor silently drops it even though config.json declaresmtp_num_hidden_layers: 1. It is excluded from quantization via re:.*mtp.*.
Acceptance rate has not been measured; the head is verified to load and generate.
Limitations
- Single evaluation corpus, and no downstream task benchmarks.
- Abliterated base. This model has had its refusal directions removed upstream; that
behaviour is inherited here and is not something quantization changes. - The abliterated calibration set is ~18% smaller than the stock one (the same length
filter kept fewer generations), so it is not perfectly matched to the stock-model builds. - Vision tower untouched (BF16); evaluated as a text model.
- MTP acceptance rate unmeasured.