license: other
license_name: swift-open-license-1.0
license_link: LICENSE
pipeline_tag: image-text-to-text
tags:

• qwen3_5
• qwen3.8
• swift
• autoround
• w4a16
• compressed-tensors
• uncensored
• abliterated
• mtp
• vllm
  base_model: ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-MTP
  base_model_relation: quantized

Swift-1.5-Qwen3.8-27B-Uncensored-W4A16-AutoRound

A W4A16 quantization of ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-MTP
(an abliterated Swift 1.5, UkisAI's reasoning-efficient
fine-tune of Qwen3.8-27B). It is built to serve on one 24 GB GPU (RTX 3090) with vLLM, MTP speculative decoding
and long context. It succeeds our Swift 1.0 build,
ultimaterex/Swift-Qwen3.8-27B-Uncensored-W4A16-AutoRound,
and was compared head-to-head against it on the same hardware and serving stack (below).

Built for syv-ai/HyperQwen on a single RTX 3090. The recipe was
chosen for that target: int8 embeddings, int4 lm_head and MTP head, and int4 linear-attention projections keep
the weights at 15.8 GB, so HyperQwen can hold a ~200k-token KV pool with MTP speculative decoding on 24 GB. On a
bigger GPU you would likely keep more of the model in bf16; see Why the heads are quantized
for what these choices cost.

Needs a patched vLLM (two small patches, see Serving). Stock vLLM fails to load it with
There is no module or parameter named 'embed_tokens.weight_packed'.

What's in the checkpoint (verified from the tensors)

Weights total 15.8 GB. The int8 embeddings and int4 heads save ~3.1 GB over the stock AutoRound layout (bf16 heads),
which goes back into KV cache on a 24 GB card. See Why the heads are quantized.

Quantization recipe

• Body: AutoRound 0.15 W4A16 g128 sym, 300 iterations (SignRound, best-iteration selection), seed 42.
• Calibration: 128 samples × 4,096 tokens. 96 samples are this model family's own chat-templated generations
  (UltraChat chat, Magicoder code, GSM8K math, mostly with thinking traces), packed; 32 are NeelNanda/pile-10k.
  The Swift 1.0 build used 128 × 2,048 tokens of pile-10k at 200 iterations. With the longer, in-distribution data,
  59 of 64 blocks found their best result after iteration 200 (the 1.0 run's blocks stopped improving between
  67 and 197).
• Heads: embed_tokens → int8 RTN; lm_head → int4 GPTQ (Hessian from 300k captured final hidden states);
  MTP → int4 GPTQ (Hessians from the MTP layers' own inputs). Hidden states were captured over 6.7M tokens that this
  checkpoint generated itself: 6,761 prompts and this checkpoint's own responses to them (thinking on/off per prompt,
  the model's default sampling), 6.7M tokens in total.
• Tooling: syv-ai/HyperQwen's prepare/ and drafter/ pipeline.

Serving

Tested with HyperQwen (vLLM 0.29.0 + its patch series) on one RTX 3090.
With plain vLLM, apply at least:

• patches/qwen3_5-embed-quant.patch: passes quant_config to the embedding (int8 embed_tokens)
• patches/qwen3_5-mtp-draft-vocab.patch: the MTP drafter scores the 40,960-row draft vocabulary

patch -p1 -d <venv>/lib/python3.12/site-packages/vllm < qwen3_5-embed-quant.patch
patch -p1 -d <venv>/lib/python3.12/site-packages/vllm < qwen3_5-mtp-draft-vocab.patch

The vLLM issue for the embedding bug is vllm-project/vllm#54304.

The config we run as our main model is HyperQwen single-user mode with SPEC=mtp CTX=long MAX_LEN=196608 PREFIX_CACHE=1 INT8_ACT=int8 (fp8 KV, a 207,530-token KV pool on a 3090, two concurrent streams).

Evaluation: Swift 1.5 build vs our Swift 1.0 build

Same RTX 3090, same serving stack (HyperQwen / vLLM 0.29.0), same settings, run back to back. "v1" is
ultimaterex/Swift-Qwen3.8-27B-Uncensored-W4A16-AutoRound.

The two models are different fine-tunes, so the perplexity and loss rows compare models, not quantization quality.
Refusal counts use a keyword detector that also fires on answers that merely mention words like "illegal" while
complying; the hand-checked row is the meaningful one. The remaining strict hits are "as an AI I can't physically …,
but here is …" deflections.

Speed (one RTX 3090, the config above)

Single runs of each test. Decode speed with speculative decoding depends on the text generated.

Why the heads are quantized

On a 24 GB card every GB of weights is KV cache. We measured the cost of the head quantization on the Swift 1.0
build (same recipe) against the same checkpoint with bf16 heads:

• int8 embed_tokens: no measurable effect (perplexity 8.2710 vs 8.2712).
• int4 GPTQ lm_head: ~0.45% perplexity, flat with context depth up to 64k (no compounding), with no extra
  unfinished runs and no reasoning loops in long-context math and compaction tests. A bf16 lm_head would cut the
  maximum context on a 3090 from 196,608 to 153,920 tokens.

On a larger GPU, bf16 heads (and bf16 linear attention) are a reasonable choice; quantizing the linear-attention
projections cost ~0.1-0.2% perplexity in the same test, growing slightly with depth.

Known limitations

• Needs the two vLLM patches above; there is no GGUF / llama.cpp version of this file.
• The generation tests reach ~40k tokens of context and up to 10 compaction cycles; sample sizes (12 episodes,
  40 problems) resolve large differences, not 1-2%.
• The abliteration is ajgazin's single-direction ablation; see their card for its method and their own refusal / KL
  measurements.

License

Swift Open License v1.0 (see LICENSE), including its commercial-use threshold (Section 5). Includes the Apache 2.0
license of the Qwen3.8-27B base model (LICENSE-APACHE-2.0) and the upstream NOTICE.

Copyright 2026 UkisAI. Swift Contribution licensed under the Swift Open License v1.0
(https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-27b/blob/main/LICENSE). Derivative of Qwen3.8-27B, Copyright 2026
Alibaba Cloud, Apache License 2.0.

Changes in this repository: the model weights of ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-MTP were quantized as
described above (AutoRound W4A16 body, int8 embeddings, GPTQ int4 lm_head and MTP head, a draft-vocabulary slice
for the MTP head), and chat_template.jinja was modified to accept OpenAI-style reasoning_effort values
(minimal → low, high / max → xhigh) instead of raising an error.