license: apache-2.0
language:
- en
- zh
library_name: transformers
pipeline_tag: text-generation
base_model: huihui-ai/Huihui-Qwen3.8-27B-abliterated
tags: - qwen3.8
- abliterated
- uncensored
- gptq
- int4
- vllm
- xpu
- intel-arc
- arc-pro-b70
- b70
- quantization
quantized_by: zrlu
Huihui-Qwen3.8-27B-abliterated-GPTQ-Int4-sym-G128-MTP-BF16 (B70)
GPTQ-INT4 quantization, purpose-built and tested for single Intel Arc Pro B70
(Xe2, Battlemage) inference via vLLM XPU on Windows Docker Desktop + WSL2 —
and for pi/opencode agent workloads on the same card.
- Model name: Huihui-Qwen3.8-27B-abliterated-GPTQ-Int4-sym-G128-MTP-BF16-B70
- Original model: huihui-ai/Huihui-Qwen3.8-27B-abliterated
(abliterated update: layers 18–51 ablated only; MTP and vision untouched),
itself a derivative of Qwen/Qwen3.8-27B (Apache-2.0). - Type / purpose-built: quantized derivative checkpoint (GPTQ-INT4), MTP (spec-decode)
heads preserved in BF16 — required by the vLLM B70 MTP4 recipe.
License
Apache-2.0 — inherited unchanged from the base model. See LICENSE and the
base model card. This is a weight-only quantization; no rights are widened.
Quantization (self-hosted, reproducible)
- Tool:
gptqmodel==7.3.2; run on a CUDA GPU (RTX 5090) with lazy checkpoint
loading; calibration: wikitext-2 (or bundled fallback text). - Contract:
bits=4, group_size=128, desc_act=false, sym=true, lm_head=false, dynamic={"-:.*mtp.*":{}}(MTP tensors stay BF16). - Input 51.75 GB (BF16) → output 18.22 GB (−64.8%), 5 safetensors shards,
config archQwen3_5ForConditionalGeneration,image_token_id 248056. - 400 quantized INT4 weight tensors + 15 preserved BF16
mtp.*tensors.
Serving (Arc Pro B70)
Served with vLLM XPU 0.27.2rc1.dev77+gac7509e2b (kernels 0.1.12.3), MTP4 +
draft-INT4 S+M1 overlay, prefix caching, qwen3_xml tool-call parser, fp8 KV.
Reference run (WSL2, Windows, self-reported):
| workload | result |
|---|---|
| decode p512/g128 (client post-first) | ~53–59 tok/s |
| concurrent N=4 decode Σ | ~141–173 tok/s |
| MTP acceptance | ~48–54% |
| 8.8 GiB KV fixed | ~206k tokens pool (2× 100k-context) |
One-click container: zrlu/qwen38-27b-arc-pro-b70:2026.08.24 on Docker Hub
(entrypoint auto-downloads this repo to /model on first start), Dockerfile and
pi-agent setup in the GitHub repo.
Usage
# standalone / eval
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("zrlu/Huihui-Qwen3.8-27B-abliterated-GPTQ-Int4-sym-G128-MTP-BF16-B70")
m = AutoModelForCausalLM.from_pretrained(
"zrlu/Huihui-Qwen3.8-27B-abliterated-GPTQ-Int4-sym-G128-MTP-BF16-B70",
device_map="auto")
Signed: zrlu. GPU: Arc Pro B70 (0xe223) + RTX 5090 (quantization only).