license: apache-2.0
base_model: JonathanColetti/Qwen3.8-27B-Uncensored
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: transformers
language:

• en
• zh
  tags:
• qwen3.8
• w4a16
• auto-round
• gptq
• marlin
• vllm
• vision
• uncensored
• abliterated

Qwen3.8-27B-Uncensored-W4A16-vision

4-bit (W4A16, group size 128) quantization of
JonathanColetti/Qwen3.8-27B-Uncensored,
with the bf16 vision tower preserved and the MTP head omitted.

18 GB on disk. Serves on 2×RTX 3090 (TP=2) with 256K context and ~12.9 GB
free for KV cache.

Attribution chain
Qwen/Qwen3.8-27B (Apache 2.0)
→ JonathanColetti/Qwen3.8-27B-Uncensored — abliteration via Heretic, 200-trial Pareto search
→ this repo — W4A16 quantization + vision-tower repack

The abliteration is not my work. What is mine here: the W4A16
quantization pipeline, the text/vision split before calibration, and the
repack that reattaches the bf16 vision tower to the quantized text stack.

Variants

Both were produced from the same quantization run; they differ only by the
849 MB model-mtp.safetensors. vLLM skips mtp.* tensors unless
speculative decoding is explicitly enabled, so the two behave identically
without it.

Deployment (verified)

Hardware: 2×NVIDIA RTX 3090 24 GB (NVLink), TP=2, GPU 0+2.

vLLM launch (0.20.2, float16 compute):

vllm serve <path> \
  --host 0.0.0.0 --port 9411 \
  --dtype float16 --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.95 \
  --max-model-len 262144 --max-num-seqs 16 \
  --enable-prefix-caching \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-processor-kwargs '{"max_pixels": 2097152}' \
  --limit-mm-per-prompt '{"image": 8, "video": 2}'

Runtime footprint (measured at startup):

Why the flags matter:

• --mm-processor-kwargs '{"max_pixels": 2097152}': without this, images
  above ~2M pixels trigger a tokenizer truncation and fail with
  Mismatch in image token count between text and input_ids. 2 M ≈ 1448×1448
  (or a long screenshot around 900×2300).
• --limit-mm-per-prompt '{"image": 8, "video": 2}': enables the video
  input path (vLLM defaults undeclared modalities to zero) and lifts the
  per-request image cap.
• --reasoning-parser qwen3: separates the model's <think> block from the
  answer body. Recommended if you consume message.content in a pipeline
  where thinking noise is unwanted; drop it if you want reasoning inline.

Throughput

Measured on the same 2×3090 during the evaluation runs:

The 16-concurrent aggregate is what you'll actually see in a real chat
front-end serving multiple sessions.

Quantization recipe

Text-only calibration on 256 samples × 2048 tokens from NeelNanda/pile-10k.

auto-round /models/<text-only-source> \
  --bits 4 --group_size 128 --format auto_round:auto_gptq \
  --nsamples 256 --seqlen 2048 --low_gpu_mem_usage \
  --ignore_layers <all linear_attn.in_proj_a/b across 64 layers>

Details, all learned the hard way:

• The vision tower is stripped before quantization and repacked afterwards.
  auto-round's is_mllm_model() detects preprocessor_config.json and
  silently switches to MLLM calibration with the wrong dataset. The file
  must be absent during quantization and present in the final
  artifact.
• linear_attn.in_proj_a / in_proj_b are excluded from quantization
  on all linear-attention layers. Qwen3.8's hybrid attention (16 full + 48
  linear across 64 layers) breaks otherwise.
• The MTP head is copied verbatim from the source checkpoint after the
  merge, never quantized (for the -mtp variant only).
• End-to-end quantization time on 2×3090 (~1 hr calibration, 3 hr AutoRound
  optimization loop, ~10 min repack).

Evaluation

Please read this before comparing to other repos' numbers. All numbers
below were produced with thinking ON (Qwen 3.8's default chat template
opens a <think> block; we did not close it). Most public leaderboards for
non-reasoning-tuned models use thinking OFF + 0-shot. On multiple-choice
sets, thinking ON typically gains 5–10 points over thinking OFF for this
model family. Do not cross-compare these scores to leaderboards that
disable thinking.

The comparisons that are valid:

1. This quantized derivative vs the unmodified Qwen3.8-27B base (same
   quantization, same harness, same seed, thinking ON). Isolates what
   abliteration + W4A16 cost end-to-end.
2. This quantized derivative vs upstream Qwen public numbers for
   Qwen3.8-27B. Same base model, different eval configuration — treat as
   a coarse sanity check, not a rank comparison.

1) Vs. our own quantized base (real, tight comparison)

Same seed, same items, thinking ON, temperature=0. Full-set accuracy and
the accuracy on the subset where thinking finished within the sampling
budget:

Read the "ex-trunc" column as the actual capability figure. MC-class
tasks used a 4096-token sampling budget which is not always enough for
Qwen3.8's default <think> block to close on hard items — those items were
scored as wrong but the failure is a sampling budget artifact, not a
capability loss. Truncation rate stays around 8–17 % per task on both
models.

Vision tower: 7/7 on a self-composed synthetic-image battery (solid
colours, shape counting, OCR, grid counting, dense colour count, 1808×4000
resize test). Identical to the base.

Tool calling: 8/11 on a mixed suite (8 standard single-tool cases + 1
multi-tool composition + 2 boundary cases where no tool should be called).
The base scores 7/11 on the same suite (this one is one point better
because it correctly routed a flight-booking case).

2) Vs. Qwen upstream public numbers (coarse sanity check)

Qwen publishes Qwen3.8-27B scores as visual-formatted tables under different
eval conditions than ours. As a sanity check that this derivative did not
collapse:

We deliberately did not run MathVision / OSWorld / BabyVision / CharXiv —
they need extra harness setup and would only add noise. GSM8K 100 % and
HumanEval 91 % (ex-trunc) are strong on their own for a 27 B W4A16 model.

3) Refusal behaviour

100 held-out prompts from
mlabonne/harmful_behaviors
(test split, same 100 as Jonathan's upstream measurement). Scoring is a
refusal-prefix regex, run with thinking OFF to match upstream's method.

Base 99 vs upstream 98 confirms our quantization did not perturb the
model's original safety layer — a difference of one prompt within the
scoring's noise floor.

Uncensored 0 vs upstream 12 deserves a note: both are refusal-rate
lower bounds (the regex catches "I'm sorry"-style prefixes, not "sure, here
is …" that then delivers a refusal). We manually poked the model with
adversarial inputs well beyond what the automated harness covered — dark,
graphic, and taboo — and could not find a topic that produced a refusal.
As the human tester put it, "I ran out of things I could think of to ask."
So the 0/100 number is consistent with what we observed in practice, but
it should be read as "we could not find a refusal in extensive manual
testing", not as a formal proof of complete removal.

The published data files in this repo include the full JSON of MMLU / CMMLU
/ C-Eval / ARC-C / TruthfulQA / GSM8K / MATH-500 / BBH / HumanEval /
IFEval / vision / tool-calling results (per-item gold + reply for
reproducibility). The 100-prompt refusal probe transcripts are
deliberately not included — they contain the full harmful-request payload
and are not something we want to redistribute even for reproducibility.
Use mlabonne/harmful_behaviors test split directly with the same seed
if you want to reproduce.

Known limitations

• Identity confusion. Asked "who are you?", the model sometimes claims
  to be Claude or GPT. Inherited from base training data. Set a system
  prompt to enforce identity if this matters.
• Refusals are reduced, not verified-removed. The upstream author
  reports 12/100 on held-out prompts; we could not reproduce any refusal on
  100 automated + extensive manual testing. But this cannot be interpreted
  as proof of exhaustive removal.
• Fine visual detail is a Qwen3.8 property, not a specialized
  vision-tower's. The tower here is the general-purpose one from
  Qwen3.8-27B, unchanged by quantization. Dedicated Qwen3-VL-* models
  will do better on OCR of dense small text and small-object counting.
• Single images above ~2M pixels must be either downsampled by the
  server (via --mm-processor-kwargs '{"max_pixels": 2097152}') or by the
  client. Without the flag they fail with a token-count mismatch.
• TruthfulQA drops 5 pp vs the base. The largest measured cost of
  abliteration in our suite. "Do not refuse harmful requests" and "prefer
  the truthful answer over a plausible-sounding one" are partly aligned
  optimization targets, and pulling on one moves the other. Expected, but
  something to be aware of.

Intended use

Local inference and research. Not intended for deployment to third parties
without your own safety layer — the same condition the upstream author
states, doubly so at 0/100.

License

Apache 2.0, inherited through the chain above. Qwen's acceptable use policy
applies to this derivative as it does to the base model.