license: other
base_model: huihui-ai/Huihui-Qwen3.5-27B-abliterated
tags:
- qwen3.5
- quantized
- w4a16
- compressed-tensors
- abliterated
- vllm
- mtp
model_type: qwen3_5
quantized_by: j-a-a-a-y
Huihui-Qwen3.5-27B-abliterated — CT-W4A16 (text-only + MTP)
4-bit weight-only quantization of huihui-ai/Huihui-Qwen3.5-27B-abliterated.
Text-only: vision encoder removed to save 0.9 GB VRAM.
MTP preserved: speculative decoding works out of the box.
Specs
| Property | Value |
|---|---|
| Base model | Qwen3.5-27B (abliterated) |
| Quantization | W4A16 (compressed-tensors, group_size=128) |
| Size on disk | 18.5 GB |
| vs GPTQ W4A16 | -1.0 GB (no vision encoder) |
| MTP head | Included (BF16, 0.85 GB) |
| Vision encoder | Removed |
| Calibration | 128 samples, Pile validation |
Usage with vLLM (RTX 5090 / 32GB)
python -m vllm.entrypoints.openai.api_server \
--model j-a-a-a-y/Huihui-Qwen3.5-27B-abliterated-CT-W4A16 \
--served-model-name qwen3.5-27b \
--dtype float16 \
--max-model-len 1024 \
--max-num-seqs 4 \
--gpu-memory-utilization 0.97 \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 5}' \
--performance-mode interactivity
Benchmarks (RTX 5090, MTP=5 + interactivity)
| Metric | GPTQ W4A16 | This model |
|---|---|---|
| Single 256 tok | 149 tok/s | TBD |
| Single 512 tok | 131 tok/s | TBD |
| Batch=4 agg | 410 tok/s | TBD |
| MTP acceptance | 50% | TBD |
| Available KV cache | 6.35 GiB | ~7.3 GiB (est.) |