license: apache-2.0
base_model: JonathanColetti/Qwen3.8-27B-Uncensored
base_model_relation: quantized
tags:

• qwen3_5
• qwen3.8
• awq
• w4a16
• compressed-tensors
• abliterated
• uncensored
• mtp
• vllm
• vision
  pipeline_tag: image-text-to-text
  library_name: transformers

Qwen3.8-27B-abliterated-AWQ-MTP

W4A16 AWQ quantization of the abliterated (refusal-removed) Qwen3.8-27B, with the MTP
speculative-decoding module preserved.

As of 2026-08-15 this appears to be the only AWQ / W4A16 build of an abliterated
Qwen3.8-27B. Other abliterated derivatives are GGUF, MLX, NVFP4, MXFP4, BF16 or INT8 —
none of which give a fast 4-bit weight-only path under vLLM on Ampere. It is also one of
only two abliterated builds of any format that preserve the MTP module.

To be clear about what this is not: AWQ builds of the base (non-abliterated)
Qwen3.8-27B do exist and several predate this one. The gap filled here is the
abliterated + W4A16 + MTP combination, not AWQ of this model in general.

• Base: JonathanColetti/Qwen3.8-27B-Uncensored (bf16 abliteration, the only abliterated 3.8 that kept the MTP module)
• Upstream: Qwen/Qwen3.8-27B (Apache-2.0)
• Method: llm-compressor 0.13.0, AWQ + W4A16_ASYM, group size 128
• Calibration: 128 samples × 1024 tokens from HuggingFaceH4/ultrachat_200k

Architecture notes

qwen3_5 is a hybrid architecture and a VLM:

Left in bf16 (not quantized): the whole vision tower (model.visual.*), lm_head,
mtp.*, and in_proj_a / in_proj_b — the 48-wide per-head scalars driving the delta rule,
where 4-bit is destructive and saves nothing. conv1d is not an nn.Linear so AWQ never
touches it.

v_proj → o_proj smoothing is deliberately skipped: GQA means v_proj emits 1024 while
o_proj takes 6144 (q_proj is doubled to 12288 by attn_output_gate), so the shapes do not
line up for a channel-wise scale.

Measured on 2× RTX 3090 (TP=2, vLLM)

All figures are single-stream (c=1), 600 output tokens, thinking off, measured over an HTTP
round-trip with the same harness — so they are directly comparable to each other.

Note on absolute numbers. Throughput on these builds is workload-dependent by roughly
30%: MTP acceptance is far higher on predictable text than on varied prose. Measured
2026-08-21 on the sibling HauhauCS AWQ build under a pinned protocol — same box, same
settings, same day — an expository prompt gives 95.9 t/s where creative prose gives 74.3 t/s.
The figures in this section were all taken in one session with one harness, so they are
internally comparable; but the prompt was not recorded, so do not quote the absolute
values against numbers measured elsewhere. The relative comparisons (W4A16 vs FP8, MTP on vs
off) are unaffected.

What MTP is actually worth here

Measured on this model, same harness, only --speculative-config changed:

That third row is the failure in trap 3 below, and it is worth internalising: a broken MTP is
slower than no MTP at all, because you pay the full drafting cost (~98 t/s of drafted
tokens) and accept none of it. It presents as a mysterious throughput regression, not as a
loading error. Disabling MTP entirely also buys you a larger KV cache (677,958 tokens,
2.59× at 262,144) if context depth matters more to you than latency.

Verified after quantization: 250,060-token needle retrieval; tool calling (correct JSON
arguments); the abliteration survived; and vision still works — the model correctly
described shapes and colours and read embedded text from a test image. The vision tower is
left in bf16, so image quality should be unchanged from the source.

Not measured: no perplexity or benchmark comparison against the bf16 source was run. W4
does cost some accuracy relative to bf16/FP8; this card does not quantify it. Behaviour was
verified, quality regression was not.

Recommended sampling settings — read this before you file a bug

Do not use greedy decoding (temperature: 0) with this model. It will emit the same
sentence over and over until it hits your token cap and never produce a stop token. This is
the classic Qwen3 + quantization degenerate-repetition mode, not a defect in the weights,
and it is fully reproducible: greedy is deterministic, so the same prompt loops the same way
every time.

generation_config.json in this repo now ships a repetition_penalty of 1.05, which is
enough to prevent it. If your stack ignores generation_config.json, set it yourself:

Measured on this build, greedy worst case, 6000-token cap:

The penalty is not free — it costs about 6% throughput. Interleaved A/B/A/B on the same
box, thinking disabled, 600-token generations, c=1: 68.4 / 67.1 t/s median with the penalty
off versus 64.3 / 63.6 t/s at 1.05. That is the price of not looping; we think it is worth
paying by default, and you can lower it to 1.02 if you would rather have the throughput.

It does not otherwise cost you anything measurable. Checked at 1.05 against penalty-off on
the tasks a repetition penalty is most likely to damage — all identical:

Do not go much above 1.10 — that is where repeated code syntax and identifiers start being
penalized.

Two traps if you try to fix this yourself

• presence_penalty will not work as a server-side default in vLLM. vLLM only carries a
  fixed whitelist out of generation config into its default sampling params —
  repetition_penalty, temperature, top_k, top_p, min_p, max_new_tokens
  (ModelConfig.get_diff_sampling_param). A presence_penalty entry is silently ignored.
• presence_penalty and frequency_penalty cannot be defaulted at all over the OpenAI
  API, because the OpenAI schema defaults them to 0.0 — clients always send them
  explicitly, so your default never applies. repetition_penalty defaults to None on the
  request and is filled from the server's defaults, which is why it is the only lever that
  survives a client setting its own temperature.

Why your client probably will not save you

Coding agents commonly pin temperature: 0 for determinism and send no penalty at all. Two
checked as of 2026-08-21: one sends no temperature when thinking is enabled but hardcodes
temperature: 0 on its sub-agent and skill paths; the other sends no temperature and no
penalty of any kind, ever. In both cases every request rides entirely on the server defaults.

Optional: chat_template_medium.jinja

This model's stock chat template defaults reasoning_effort to xhigh, which spends the
whole budget thinking and returns an empty answer on a large fraction of requests. The
default chat_template.jinja in this repo is unmodified upstream — we did not silently
change the behaviour of a redistributed artifact.

For convenience an opt-in copy is included as chat_template_medium.jinja, byte-identical
except for one line, which defaults reasoning_effort to medium instead:

-{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
+{%- set resolved_reasoning_effort = reasoning_effort|default('medium') %}

Use it with --chat-template chat_template_medium.jinja, or keep the stock template and pin
the value per-request with --default-chat-template-kwargs as shown below. Both work; the
flag merges per-key, so clients sending their own chat_template_kwargs stay covered.

Running it

Needs a vLLM recent enough to register Qwen3_5ForConditionalGeneration and the
Qwen3_5MTP proposer. This was validated on vllm/vllm-openai:nightly (v0.20.2rc1.dev129,
compressed-tensors 0.15.0.1 in-image — it reads the 0.18.0 config fine).

Minimal

vllm serve twolven/Qwen3.8-27B-abliterated-AWQ-MTP \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

The full production config (2× RTX 3090, single user)

This is what produced the numbers in the table above.

vllm serve /path/to/Qwen3.8-27B-abliterated-AWQ-MTP \
  --served-model-name qwen3.8-27b-abliterated \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 4096 \
  --kv-cache-dtype fp8 \
  --performance-mode interactivity \
  --mm-encoder-tp-mode data \
  --disable-custom-all-reduce \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "medium"}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode": "PIECEWISE"}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

With NCCL_P2P_DISABLE=1 in the environment.

Why these particular flags — each one is load-bearing on this hardware:

⚠️ Pin reasoning_effort — the default will truncate your answers

The chat template accepts reasoning_effort of xhigh / medium / low, and defaults to
xhigh:

{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}

xhigh is pathological for general serving. Measured over 4 hard reasoning prompts with a
16,384-token budget:

At xhigh, half the requests spent the entire budget thinking and returned an empty
answer (finish_reason: length). Note low is counterintuitively worse than medium —
it thinks ~58% more — so medium is the sweet spot, not merely a midpoint.

This is not a repetition loop and not a quantization artifact. A truncated xhigh trace
analysed at 675/676 unique sentences, with only 1.7% of positions inside any repeated 8-gram
and 0% repetition in the final 400 words — the model is generating novel reasoning right to
the cutoff. Re-running an identical truncated prompt with a 40,000-token budget terminated
cleanly (stop, 11,756 thinking + 2,913 answer). The failure mode is variance: xhigh's
thinking length straddles typical budgets, so the same prompt sometimes finishes at 14k and
sometimes blows past 16k and returns nothing.

Pin it server-side:

--default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "medium"}'

Clients can still opt up per request with
chat_template_kwargs: {"reasoning_effort": "xhigh"} — just give them a budget of 40k+ if
they do. Thinking can be turned off entirely with {"enable_thinking": false}.

Tool calling caveat

If you enable --tool-call-parser qwen3_coder, vLLM's parser has a streaming bug where the
<tool_call> tag is emitted as prose and subsequent SSE chunks go silent. We run a small
patch that buffers <tool_call> tokens until <function= appears within 64 characters.
Patch and details: club-3090 →
models/qwen3.6-27b/vllm/patches/local/. Non-streaming tool calls are unaffected.

Sanity checks after it boots

# should report ~560,900 tokens and 2.14x at 262,144
grep -E "GPU KV cache size|Maximum concurrency" <container logs>

# MTP must actually load — if you see "not found in params_dict" the drafter is
# running on random weights and acceptance will be 0% (see trap 3 below)
grep "Detected MTP model" <container logs>

# after some traffic, acceptance length should be ~2.9 of 3
grep "SpecDecoding metrics" <container logs>

The compose files, the quantization script, and the deployment notes this model came out of
live in club-3090.

If you reproduce this — four things that will bite you

1. AWQ mappings must be generated per layer. match_modules_set() accumulates matches
   until every pattern fires, so a generic re:.*input_layernorm$ → q/k/v mapping piles up
   four smooth layers across three consecutive linear_attention blocks (which have no
   q_proj) and aborts with "AWQ needs to match a single smoothlayer". Emit one mapping per
   layer index — 192 over 64 layers.
2. transformers erases Qwen3_5GatedDeltaNet.forward's signature. Its
   force_accelerate_hooks decorator is def wrapped(self, *args, **kwargs) with no
   functools.wraps, so inspect.signature() reports (*args, **kwargs); AWQ's arg cache
   then stores {'args': …} and replay dies with "missing 1 required positional argument:
   'hidden_states'". Recover the inner function from the wrapper closure and re-attach
   __signature__.
3. MTP needs both an index entry and an ignore rule. save_pretrained writes a single
   model.safetensors with no index, so a copied-in MTP file is never opened, the drafter
   loads random weights, and you get 0% draft acceptance — which presents as a speed bug
   (~33 t/s) rather than a load bug. Write a model.safetensors.index.json covering both
   files, and add re:.*mtp.* to quantization_config.ignore or vLLM builds MTP as quantized
   and demands weight_packed. After changing that, clear the torch.compile cache or a stale
   AOT graph throws KeyError: 'weight_packed'.
4. AWQ will OOM 24 GB at 256×2048 calibration. Use offload_device="cpu" on
   AWQModifier and drop to 128×1024.

Author

Quantized by Todd Wolven - Lead AI Software Developer and open-source GenAI engineer.

Other projects and writeups | GitHub | Hugging Face

License & attribution

Apache-2.0, inherited from Qwen/Qwen3.8-27B. The abliteration is
JonathanColetti's; this repo contributes only the quantization.

This model is uncensored. Refusal behaviour has been removed by the upstream abliteration
and that property survives quantization. You are responsible for how you use it.