base_model: huihui-ai/Huihui-Qwen3.8-27B-abliterated
tags:
- gptq
- compressed-tensors
- quantized
- vllm
license: apache-2.0
huihui-qwen3.8-27b-abliterated-gptq
GPTQ 4-bit (W4A16) quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated, produced with llm-compressor.
⚠️ Use at your own risk. This is an unofficial, community-produced quantization of an abliterated model — the base model has had its built-in safety refusal behavior deliberately removed. Quantization was not validated for output quality, safety, or correctness beyond "it loads and generates text." No warranty of any kind is provided. You are solely responsible for how you deploy and use this model, and for complying with the original Qwen license and any applicable laws in your jurisdiction.
Quantization details
| Method | GPTQ, scheme W4A16 |
| Group size | 128 |
| Symmetric | yes |
| Activation order | static |
| Format | compressed-tensors (pack-quantized) |
| Ignored modules | lm_head, non-Linear submodules (norms, the linear_attn container modules themselves) |
| Calibration dataset | open_platypus, 512 samples, max_seq_length 2048 |
| Weights size | ~17.6 GB (down from ~55.6 GB in bf16) |
lm_head and the embedding table are kept in bf16 (not quantized).
Architecture notes — read before deploying
- Base model is
Qwen3_5ForConditionalGeneration, a hybrid architecture: most layers use Gated DeltaNet (linear attention), with regular full attention every 4th layer (64 layers total, 48 linear-attention / 16 full-attention). This is a very new architecture family (Qwen3.5) — make sure your inference stack actually supports it before assuming it will "just work". - vLLM: confirmed present in the model registry as of vLLM
v0.27.0+ (Qwen3_5ForConditionalGeneration→qwen3_5). Use vLLM>= 0.28.0if possible.--trust-remote-codeis required. - This is a vision-language / omni model, not text-only (
image_token_idis set,language_model_only: falseinconfig.json). The vision tower loads inbf16alongside the quantized text backbone even if you only ever send text prompts — budget a few extra GB of VRAM beyond the 17.6 GB weight size for it. - AWQ quantization was attempted first and failed on this architecture —
llm-compressor's AWQ smoothing step doesn't handle the customforward()signature of the Gated DeltaNet module (Qwen3_5GatedDeltaNet.forward() missing 1 required positional argument). GPTQ (forward-hook based, no manual sample replay) worked cleanly across all layers, hence GPTQ was used here instead of AWQ.
Suggested serving config
vllm serve mma404/huihui-qwen3.8-27b-abliterated-gptq \
--trust-remote-code \
--quantization compressed-tensors \
--max-model-len 16384 # size to your actual workload; this model supports up to 262144
Fits comfortably on a single 24-48GB GPU (e.g. RTX A6000, RTX 4090, L40S) at moderate context lengths; scale --max-model-len and GPU size up for longer contexts or higher concurrency.