license: apache-2.0
library_name: vllm
base_model:
- huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated
tags: - quantized
- awq
- w4a16
- q4
- mtp
- abliterated
- moe
quantization: - awq
Huihui-Qwen3.6-35B-A3B-abliterated — AWQ Q4 (W4A16, group 128)
AWQ-quantized int4 (Q4) version of
Huihui-Qwen3.6-35B-A3B-abliterated
(35.95B-parameter MoE, ~3B active per token, hybrid linear attention), quantized
for the darkkernel/vllm-gfx1030
inference image (vLLM v0.19.1, ROCm, RDNA2 / gfx1030 — RX 6800/6900 XT and
Radeon Pro V620).
Quantization
| Scheme | W4A16 — int4 weights, 16-bit (bf16) activations |
| Group size | 128 |
| Algorithm | AWQ (activation-aware), duo scaling |
| Tool | llm-compressor (AWQModifier, scheme W4A16) |
| Calibration | 512 samples, HuggingFaceH4/ultrachat_200k (chat template, 2048 context) |
| Quantized | all Linear layers — decoder stack (incl. 256-expert MoE) and vision tower |
| Kept bf16 | embeddings, RMS norms, lm_head, Conv layers, and the mtp.* speculative-decoding module (19 tensors, stored as bf16 — documented llm-compressor fallback; vLLM aligns draft-model quantization automatically) |
MTP speculative decoding
The checkpoint includes the quantized MTP (multi-token prediction) module
(mtp.*, 1 layer). vLLM v0.19.1 detects it automatically
(qwen3_5_moe → Qwen3_5MoeMTP), and the draft-model quantization is aligned
to the target model automatically — no extra flags needed beyond--num-speculative-tokens.
Serving (gfx1030)
docker run -d --name vllm-server \
--device=/dev/kfd --device=/dev/dri \
--security-opt seccomp=unconfined \
--ipc=host --shm-size=16g \
-e HIP_VISIBLE_DEVICES=0 \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
-p 8000:8000 \
darkkernel/vllm-gfx1030:v0.19.1-rocm7.2 \
vllm serve mostwanted002/Huihui-Qwen3.6-35B-A3B-abliterated-awq-q4 \
--max-model-len 8192 --host 0.0.0.0 --port 8000 \
--num-speculative-tokens 1
curl -s http://localhost:8000/v1/completions -H 'Content-Type: application/json' \
-d '{"model":"mostwanted002/Huihui-Qwen3.6-35B-A3B-abliterated-awq-q4",
"prompt":"The capital of France is","max_tokens":24}'
RDNA2 has no native FP8, which is why this checkpoint is int4 (Q4) with
16-bit activations rather than an FP8/MXFP4 scheme.