license: apache-2.0
library_name: transformers
tags:

• vllm
• intel-xpu
• int4
• autoround
• gptq
• mtp
• speculative-decoding
• qwen
• uncensored

Qwen3.8-27B-Uncensored INT4 AutoRound (MTP-preserved)

An INT4 (W4A16) quantization of an uncensored/abliterated 27B dense Qwen3.8
variant, built for fast local serving on Intel Arc Pro B70 GPUs with vLLM XPU.

Two deliberate choices distinguish this checkpoint from a stock INT4 quant:

1. The MTP (multi-token prediction) head is preserved and quantized, not
   stripped. Speculative decoding works out of the box with vLLM's
   qwen3_5_mtp method — no separate draft model or weights surgery needed.
2. The linear-attention in_proj_a / in_proj_b projections are kept in
   FP16. Quantizing those two projection families to INT4 g128 measurably
   degrades exact-answer behavior (e.g. deterministic code-evaluation prompts
   return wrong values). Keeping them FP16 costs a few percent of decode
   bandwidth and preserves target-model quality. Verify any other INT4 build
   of this model against a deterministic canary before trusting it.

Quantization

The checkpoint is directly servable as GPTQ INT4 (Marlin /
XPUwNa16LinearKernel path in vLLM).

Measured performance (Intel Arc Pro B70)

Cold-response measurement discipline: fixed realistic prompt suite, each
prompt run once, prefix/prompt caching disabled, cached_tokens=0 on every
request; primary metric is the median inter-token rate over tokens 1-100
after TTFT.

Quality gates on this exact checkpoint: arithmetic, deterministic code
evaluation, copy, factual, JSON-schema, long-context needle, and
repeat-stability checks all pass; all exact canaries are identical to the
FP16-head reference.

Notes on interpretation: warmed, repeated-prompt, same-shape benchmarks of
this stack read much higher (80+ on a single card); those numbers describe
repeat traffic, not fresh requests.

Serving (vLLM XPU)

vllm serve /path/to/this/model \
  --quantization gptq \
  --dtype float16 \
  --tensor-parallel-size 2 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9 \
  --enable-prompt-tokens-details \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'

• Single-GPU serving needs no tensor parallelism flags; the model fits in one
  32 GB B70 with FP16 KV at moderate context.
• FP8 KV cache (--kv-cache-dtype fp8) works and is the capacity option for
  128K context on one card; FP16 KV decodes slightly faster.
• The MTP head accepts num_speculative_tokens 1-4. Depth 3 was the best
  measured trade-off on fresh-response workloads; depth 4 can win on highly
  predictable content (acceptance is workload-dependent — check
  vllm:spec_decode_num_accepted_tokens_total against
  spec_decode_num_draft_tokens_total on your own traffic).
• Optional experimental lever: an INT8-quantized LM head (W8A8) served via a
  patched vllm-xpu-kernels build adds ~15% decode on this model class; the
  stock FP16 head path is what the 81.8/57.5 rows above use.

Chat template

Ships the stock Qwen3.8 chat template (chat_template.jinja; reasoning and
tool-call blocks follow the base model's conventions). Note for serving:
vLLM's thinking_token_budget (or an equivalent hard cap) is the reliable
way to bound reasoning length; prompt-level effort instructions are a hint
the model can ignore on hard prompts.

Limitations

• This is an uncensored model: it will answer prompts that safety-tuned
  models refuse. You are responsible for how you use it; review generated
  content before relying on it.
• INT4 quantization trades some quality for speed and size. The preserved
  FP16 projection families mitigate the worst regression we measured, but
  outputs are not bit-identical to the BF16 original, and stylistic casing
  differences on short answers have been observed.
• Benchmarks above are single-stream; concurrent serving aggregates higher.