license: apache-2.0
base_model: huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated
tags:

• qwen3_5
• coder
• gptq
• int4
• w4a16
• vllm
• xpu
• intel-arc
• mtp
• gdn
• text-generation
• conversational
• en
• zh
  library_name: vllm

Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-GPTQ-Int4-sym-G128-MTP-BF16

GPTQ Int4 W4A16 (symmetric, group-128, desc_act=False, lm_head=False)
quantization of
huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated
(dense Qwen3.8-27B coder, 64 layers of hybrid linear/full attention), built for the
vLLM XPU backend on Intel Arc — validated on an Arc Pro B60.

The recipe mirrors
SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16
field-for-field, so this is a drop-in swap of that checkpoint onto the Coder390 base
rather than a new quantizer configuration. As of 2026-10-18 it is the only GPTQ
quantization of Coder390 on the Hub — the family has GGUF, MLX, EXL3, NVFP4,
compressed-tensors W4A16 and MXFP4 builds, but no GPTQ until this one.

Contents

• Quantized (400 modules → qweight int32 + scales/qzeros): all 64 dense
  mlp.{gate,up,down}_proj, all 16 full-attention self_attn.{q,k,v,o}_proj, and all
  48 linear_attn.{in_proj_qkv,in_proj_z,out_proj}. in_proj_a/in_proj_b are
  skipped, as in the reference recipe.
• BF16: embeddings, lm_head, every norm, A_log/conv1d/dt_bias, the full
  27-block visual tower, and the MTP draft block (mtp.*, 15 tensors) — held out of
  quantization with dynamic={"-:.*mtp.*": {}} and stored unquantized inside shard 5,
  so speculative decoding needs no separate draft file.
• text_config.mtp_num_hidden_layers is 1 (as shipped upstream).

2,399 tensors / 5 shards / 18.24 GiB · SHA256SUMS included.

Structurally identical to the reference checkpoint (same 2,399 tensors, same per-dtype
element counts: F16 3,219,819,760 · I32 3,067,527,168 · BF16 424,699,392), which is
what makes the two directly comparable below.

Recipe

The calibration is the one deliberate deviation from the classic GPTQ budget
(128 × 2048 = 262,144): on the 45 GB-RAM host used here the larger pile terminated the
job every time (GPU engine reset, or a host OOM-kill), and 64 × 1024 completed cleanly.
The reference checkpoint does not record how it was calibrated, so token-set parity was
never possible anyway. If you re-quantize, note that a different calibration changes the
Hessians and therefore the weights — it is a different artifact, not a re-run.

Measured serving

Measured on Intel Arc Pro B60 (24 GB, Battlemage), i5-12400F host, GPU power capped
at 180 W, vLLM 0.30.0+xpu, --dtype float16 --kv-cache-dtype fp8 --language-model-only,
MTP speculative decoding k=4, prompt 512 / generation 128 tokens:

MTP is worth having: with the draft disabled the same build measured 7.987 tok/s
against 16.620 tok/s at k=4 — 2.08× — on an MXFP4 leg of the same experiment.
Note the acceptance ratio is an upper bound on the gain, never the gain itself: the
speculative step costs 1.361× more per step here (125.2 → 170.5 ms).

Long context (single stream, 12,288-token ceiling): prefill stays linear at
~1,372 tok/s through 4,096 tokens (marginal 1,268 tok/s against the fitted line,
R² = 0.9986), while decode degrades gently — 40.200 tok/s @512 → 37.315 tok/s @12,000
(−7.2 %), TTFT 0.463 → 9.539 s. With --kv-cache-dtype fp8 the KV cache is 1.82 GiB,
i.e. ~17,630 tokens of capacity at these settings.

Measured quality

15/16 (93.8 %) on a 16-prompt task suite spanning math, code (executed), instruction
constraints and fact/trap questions. Treat that as ±1 item: one item
(format_no_letter_e) converges on some server instances and loops to the token cap on
others at temperature=0, deterministically within an instance but not across them, so a
single-item delta between builds is not signal.

Two harness traps this checkpoint sets, both worth knowing before you benchmark it:

• Always pass --reasoning-parser qwen3. The engine emits </think> (added token
  248069. but the detokenizer suppresses it from content and leaves reasoning
          null, so without the parser the whole reasoning trace arrives concatenated in front of
          the answer with no delimiter. vLLM's own 16-prompt quality harness scores 8/16
          that way versus 15/16 with the flag — a plausible-looking 50 % scored on the
          model's private musings.
• Quality items can flip between server instances even at greedy decoding (see above).
  Re-run a lone failure before believing it.

Serving

vllm serve Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-GPTQ-Int4-sym-G128-MTP-BF16 \
  --served-model-name coder390-gptq \
  --quantization gptq --dtype float16 --kv-cache-dtype fp8 \
  --max-model-len 12288 --max-num-seqs 1 --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.93 --language-model-only \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"mtp","num_speculative_tokens":4}'

Notes:

• --language-model-only skips the visual tower, which is what the benchmarks above
  used. The tower is intact and BF16 (333 tensors) if you need vision.
• MTP resolves automatically: the draft is the BF16 mtp.* block in shard 5, activated
  by B70_MTP_BF16_DRAFT=1 on this stack and by the compressed-tensors ignore list.
• On a cold container the first engine init can fail with
  ValueError: ... KV cache is needed, which is larger than the available KV cache memory (0.18 GiB) because torch.compile's transient pool eats the KV budget; the
  retry succeeds against a warm compile cache. Run the server under
  --restart unless-stopped (or restart the same container once) and expect ~220 s to
  healthy rather than treating the first failure as fatal.
• Decode throughput on this host is bimodal (~40.2 vs ~48.5 tok/s, ~20 %), with the
  mechanism unidentified and step time — not acceptance — responsible. Compare rates
  only within a single session, and report a median and a spread rather than one number.

Relation to the MXFP4 sibling

alpha2beta/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-MXFP4
is the same base model quantized to MXFP4 (compressed-tensors mxfp4-pack-quantized,
W4A4). Same tensors held in BF16, same folder layout, same 5-shard/16-shard family
naming. On the same host MXFP4 is markedly slower than GPTQ at batch 1 — W4A4 dynamic
activation quantization and a non-fused FP4 GEMM, multiplied by eager execution — which
is why this GPTQ build exists as the throughput option.

Provenance

Built with gptqmodel 7.6.0 on
vLLM 0.30.0+xpu / torch 2.13.0+xpu / transformers 5.16.1, driving the Arc Pro B60 as the
quantization accelerator, with offload_to_disk=True (meta-device shell + module-by-
module streaming) because the 52 GB BF16 source does not fit in the 45 GB host RAM.

Base weights: huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated
(BF16, Apache-2.0). Reference schema: SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16.

Full build record, attempt log, benchmarks and the operating guide:
alpha2beta/vllm.xpu @ b60 —
see REPRODUCIBILITY.md (§6, §7 deviation 8, §8 known issues 1/11/12/13) and
GPTQ_quantization.md.