license: other
license_name: swift-open-license-1.0
license_link: https://ukisai.com/contact
base_model: d0xin/Swift-Qwen3.8-27B-Uncensored-BF16
base_model_relation: quantized
library_name: transformers
language:

• en
• zh
  pipeline_tag: image-text-to-text
  tags:
• gptq
• int4
• uncensored
• abliterated
• mtp
• speculative-decoding
• intel
• arc
• intel-arc
• xpu
• vllm
• qwen3.8
• swift

Swift-Qwen3.8-27B-Uncensored-GPTQ-Int4-sym-G128-MTP-BF16

GPTQ-Int4 quantization of d0xin/Swift-Qwen3.8-27B-Uncensored-BF16, built for
Intel Arc / vLLM XPU with the MTP head preserved in BF16 so native
speculative decoding still works.

The recipe is a reproduction of kernelogic/Qwen3.8-27B-Uncensored-GPTQ-Int4-sym-G128-MTP-BF16,
applied to the Swift lineage instead of the JonathanColetti one. Same quantizer
version, same config, same MTP handling.

Why this exists

Quantizing this model naively breaks speculative decoding. The 15
mtp.* tensors are the draft head; if they get quantized along with everything
else, draft acceptance collapses and you lose roughly half your decode speed.
The fix is one line of quantize config:

dynamic={"-:.*mtp.*": {}}   # exclude mtp.* from quantization -> stays BF16

The result has 400 quantized weight tensors (I32) +
15 preserved BF16 MTP tensors.

There is a second, less obvious requirement. transformers' Qwen3_5 declares
_keys_to_ignore_on_load_unexpected = [r"^mtp.*"], so the draft tensors are
dropped at load and never become modules — the dynamic exclusion on its own
emits a checkpoint with no draft head at all, silently. What writes them back is
gptqmodel's out_of_model_tensors = {"prefixes": ["mtp"]}, which re-reads them
from the source safetensors at save time. That is also why they stay BF16 while
the body is fp16: they are copied bytes, never converted.

Provenance

Qwen/Qwen3.8-27B                          (base)
  └─ ukisai/Swift-Qwen3.8-27b                 (Swift, efficient-thinking finetune)
       └─ d0xin/Swift-Qwen3.8-27B-Uncensored-BF16
            (uncensored, rank-1 directional ablation, BF16)
              └─ this repo                             (GPTQ-Int4, 20 GB)

Quantization

Group size is 128, verified from the tensor shapes
(qweight.rows * 8 / scales.rows) rather than taken from the config's own
declaration. lm_head and the vision tower (333 tensors) are left
unquantized.

Measured performance

llama-benchy --pp 4096 --tg 256 --depth 0 across concurrency levels 1 2 4 8, prefill 2350.21 tok/s at c1. 48 SpecDecoding samples over the run.

Prefill is flat across concurrency and decode is not. One stream already saturates prefill; decode scales sublinearly and turns over once prefill/decode interference dominates. Per-request decode is the number a single user experiences.

Per-position acceptance: 0.716, 0.504, 0.367, 0.307, 0.248. Judge any MTP depth change on this curve, never on tok/s -- a decay ratio near 0.8 pays for the extra drafter pass and near 0.68 is break-even.

Serving (vLLM XPU)

vllm serve /model \
  --quantization gptq --dtype float16 \
  --kv-cache-dtype fp8 \
  --max-num-batched-tokens 8192 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --chat-template /model/chat_template.jinja

Qwen3.8 MTP on XPU needs a patched vLLM. The build used to produce and serve
this checkpoint is at
github.com/greglechin/vllm-xpu; the
Intel Arc Pro B70 cookbook
is the other published route. Context length, --tensor-parallel-size and
--gpu-memory-utilization are deliberately omitted above because they depend on
your card count and VRAM — size them from your own boot log.

Three things that will bite you

1. MTP depth is not free, and tok/s is the wrong way to judge it. Read the
   per-position acceptance curve from the engine's SpecDecoding metrics, not
   the run mean: a decay ratio around 0.8 pays for the extra drafter pass and
   around 0.68 is break-even. Changing depth typically moves throughput by less
   than boot-to-boot variance, so a single A/B run measures nothing.
2. Prefix caching depends on your engine build. Qwen3.8 is a hybrid
   GDN/Mamba architecture and upstream vLLM does not declare
   supports_mamba_prefix_caching for it, which is why the reference checkpoint
   advises disabling it (measured there: 3,036 queries → 0 hits). With the
   mamba-align patches and --mamba-cache-mode align it does work — 84.8–85.3%
   hit rate measured on the serving stack above. Check your own hit-rate metric
   rather than assuming either outcome.
3. --kv-cache-dtype fp8 is effectively required on 32 GB cards at long
   context; fp16 KV does not fit.

Limitations — please read

• No quality evaluations were run on this quantization. No perplexity
  comparison against the BF16 source, no coding or reasoning benchmarks, no
  quantitative refusal-rate testing. What was verified is structural: tensor
  counts and dtypes, the preserved BF16 MTP head, complete quantized-module
  tensor sets, group size derived from shapes, and an unquantized lm_head and
  vision tower. If you need quality guarantees, measure before relying on it.
• Calibration used general web text with no code. If coding quality matters
  to you, a code-inclusive calibration set would likely be better. This is
  inherited from the reference recipe, which names the same limitation.
• Uncensoring is inherited, not verified here. All refusal-removal
  properties come from the upstream ablation (d0xin/Swift-Qwen3.8-27B-Uncensored-BF16
  reports 0/100 refusals on its own fixed evaluation); this repo only changes
  numeric precision. Quantization was not evaluated for its effect on that
  behaviour in either direction.
• The upstream's capability evidence is thin, and it stacks with the point
  above. Its INTELLIGENCE_VALIDATION.json reports 298 paired samples with a
  95% CI of [−1.68, +5.03] pp — so a ~4% relative capability loss sits
  inside the interval and is not excluded. McNemar there operates on only 27
  discordant pairs, its MATH-500 arm scores 8% on both models (a broken
  harness cannot detect damage, and it supplies a third of the samples), and the
  agentic, tool-calling, multimodal and long-context capabilities the upstream
  card claims to preserve are not evaluated at all. Measure on your own workload
  before trusting either layer.
• This is an uncensored model. It will attempt requests an aligned model
  declines. You are responsible for how you use it.

Licence

Not Apache-2.0. These weights inherit the Swift Open License v1.0 from
ukisai/Swift-Qwen3.8-27b. Personal, research, educational, evaluation
and commercial use are free for individuals and organizations with annual
recurring revenue, including affiliates, of up to US$1,000,000. Above that
threshold, commercial use requires a separate Swift Enterprise License — contact
UkisAI.

Credits

• Qwen — base model
• UkisAI — Swift finetune
• d0xin — directional ablation
• kernelogic/Qwen3.8-27B-Uncensored-GPTQ-Int4-sym-G128-MTP-BF16 — the quantization recipe this reproduces
• gptqmodel — quantization