license: apache-2.0
base_model: orcarouter/Qwen3.8-27B-Uncensored
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- gptq
- int4
- qwen3.8
- intel-arc
- xpu
- vllm
- abliterated
- uncensored
Qwen3.8-27B-Uncensored GPTQ int4 (asymmetric)
An int4 quant of orcarouter/Qwen3.8-27B-Uncensored
(an abliterated finetune of Qwen/Qwen3.8-27B by OrcaRouter). I made it to run on a single Intel Arc Pro B70.
It's asymmetric int4, group 128, GPTQ with an MSE clip search, stored in standard GPTQ v1 format (sym: false).
The vision tower, the MTP head and the lm_head stay BF16.
Safety: this inherits the base model's removed refusal behaviour. It has no meaningful built-in guardrails.
Read the base model card for its intended-use terms.
Use it responsibly and within the Apache-2.0 license.
Quality vs the BF16 base
KL divergence over the top-20 logits on held-out 2048-token windows (lower is better):
| private chat | code | technical prose | all | top-1 agree | |
|---|---|---|---|---|---|
| symmetric GPTQ int4 g128 (usual recipe) | 0.038 | 0.078 | 0.096 | 0.069 | 91.9% |
| this quant | 0.018 | 0.040 | 0.060 | 0.038 | 94.1% |
The served version (fp8 lm_head, see the cookbook) measures KL 0.039 end to end. On the arithmetic canary from the B70
cookbook it scores 5/6 vs 4/6 for symmetric GPTQ (miss: sum(i*i for i in range(4)) -> 30). That canary is only 6
greedy near-tie questions, so treat it as noise. On the stock model a 100-question version showed no real difference
between quants.
Running it
Built for and tested on Intel Arc Pro B70 + vLLM XPU v0.31.0 with the launcher and patches in
riceharvest/b70-qwen38-cookbook. That setup gives 262K
context with vision, MTP k=4, and fp8 KV on one card.
hf download darioooooo0o/Qwen3.8-27B-Uncensored-GPTQ-Int4-asym --local-dir qwen38-uncensored-b70
MODEL_DIR=$PWD/qwen38-uncensored-b70 ./serve.sh # from the cookbook repo
On a B70 at 275 W, one request at a time. These were measured with a draft vocab tuned on my own traffic; the cookbook's
public vocab gives the same ms/step and slightly different tokens/step:
| Decode, short prompts (greedy) | 64-128 tok/s depending on content (35 ms per MTP verify step) |
| Decode at 65K / 100K context | ~79 / 65-70 tok/s greedy, ~64 sampled (43 / 46 ms per step) |
| Prefill | ~2050 tok/s at 6K, ~1450 at 74K |
| Context | 262,144 tokens, ~320K-token KV pool |
Other runtimes: the format is standard asymmetric GPTQ v1, so GPTQModel / transformers / vLLM on CUDA should
load it. I only tested on Intel XPU. Stock vLLM XPU v0.31 does not load asym GPTQ correctly without the
cookbook's patch_asym_gptq.py.
How it was made
- Quantizer:
b70quant.pyfrom the cookbook. GPTQ (damp 0.01, act-order off), asym uint4 with per-group
scale + zero, and a clip search over 11 ratios minimizing |err|^2.4. One layer at a time on the B70, ~1 h. - Calibration: 128 x 2048 tokens. A mix of private chat samples (not released), reasoning/agentic text, code and
technical prose, disjoint from the eval windows. - Not quantized: lm_head, MTP layers, vision tower, the GDN
in_proj_a/in_proj_b. - Export: GPTQ v1 (qzeros store z-1). The single group with z=0 was set to z=1 (one weight changed). It was checked
against the raw quant using GPTQModel's dequant semantics.
Credits: Qwen team (base model), OrcaRouter (finetune),
SergiioB (B70 vLLM cookbook this builds on).