license: apache-2.0
base_model: DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
base_model_relation: quantized
library_name: transformers
pipeline_tag: image-text-to-text
tags:

• qwen3_5
• autoround
• w6a16
• int6
• 6-bit
• compressed-tensors
• vllm
• humming
• mtp
• vision
• uncensored

Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU — W6A16 AutoRound

6-bit weight-only quantization (W6A16, int6 symmetric, group_size 128) of
DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
using AutoRound via llm-compressor 0.14

• auto-round 0.15.1.

• Quantized: all language-model Linear layers (400 modules: attention, MLP and GatedDeltaNet
  projections) → compressed-tensors pack-quantized.
• Preserved in BF16 (untouched, bit-identical to the base model):
  • Vision tower (model.visual.*, 333 tensors)
  • MTP / Multi-Token Prediction head (mtp.*, 15 tensors) — enables speculative decoding
  • lm_head, embeddings, norms, conv1d, and the GatedDeltaNet in_proj_a/in_proj_b projections
    (output dim 48 < group_size 128)
• Calibration: 128 samples × 2048 tokens from
  neuralmagic/LLM_compression_calibration
  with the model's own chat template, 200 AutoRound iterations per layer.
• Size: ~25.5 GB (base BF16: ~52 GB).

Quality

Perplexity measured on 50 held-out texts from wikitext-103, 16,007 evaluated tokens.
Deliberately not the calibration set, which would flatter the result. Same texts, same
tokenization (identical token count for both), same method (raw text, no chat template),
both models served by vLLM:

This is above the ~5% degradation usually accepted for weight-only quantization, so treat the
LM fidelity as degraded more than a typical quant. Read this number with the caveats below rather
than as a verdict on 6-bit quantization in general.

How this compares to W4A16

A sibling run on a different but architecturally identical checkpoint
(IstroSec/ThinkingCap-Qwen3.8-27B-abliterated) used the same protocol and recipe at 4-bit and
measured BF16 8.1133 → W4A16 8.8426 = +8.99%.

The two BF16 baselines are close (8.09 vs 8.11), so the deltas are roughly comparable and suggest
6-bit recovers about 1.6 percentage points over 4-bit. This is suggestive, not a controlled
comparison: the checkpoints differ, and perplexity sensitivity to quantization is a property of
the individual model. To isolate the bit-width effect you would need to quantize this checkpoint
at both widths with everything else held fixed.

If you need tighter fidelity

The first knob to turn is more AutoRound tuning, which is what the upstream guidance in this
project recommends: iters=400 and nsamples=256 (roughly double the wall-clock). The recipe is
also bit-width-agnostic, so producing a W8A16 variant of this same checkpoint is a one-flag change
if 6-bit turns out not to be worth the tradeoff.

Note that perplexity measures language-modelling fidelity only; it is not a substitute for
task-specific evaluation. Short side-by-side chat samples between this checkpoint and the BF16
original read as equivalent — perplexity averages small probability shifts over every token and is
the more sensitive instrument here.

Serving notes — Humming, not Marlin

This is the most important operational difference from the W4A16 builds. For 6-bit weights
vLLM uses the HummingLinearKernel (CompressedTensorsWNA16 with scalar_types.uint6b32), not
Marlin: Marlin only implements uint4b8 and uint8b128. Confirmed on load:

Using HummingLinearKernel for CompressedTensorsWNA16

The Humming kernels are JIT-compiled, so expect a slower first load and different (generally
lower) throughput than the Marlin-served W4A16 builds. Measured on 8×RTX 3090 (cold JIT, TP2,
2-prompt smoke test, enforce_eager): ~28 tok/s plain decode, ~45 tok/s with MTP speculative
decoding. These are cold-start floor numbers from a 2-prompt run, not a throughput benchmark.

Usage (vLLM ≥ 0.28)

vllm serve DoktorMincs/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W6A16-AutoRound \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --reasoning-parser qwen3 \
  --enable-prefix-caching \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

• Vision inputs work normally (image/video); the vision tower runs in BF16.
• The MTP draft head loads from the same checkpoint for speculative decoding.
• Validated on vLLM 0.28: text, MTP speculative decoding and vision all confirmed, 400 packed
  modules, in_proj_a/b correctly left in BF16.
• This is a thinking model: with a small max_tokens the whole budget can be consumed by the
  reasoning trace, leaving the visible answer empty. Use max_tokens ≥ 1024 and prefer the chat
  endpoint over raw completion.

Notes

• Architecture: Qwen3_5ForConditionalGeneration (hybrid: 64 layers, 3:1 GatedDeltaNet
  linear-attention : full-attention, 27-block ViT, 1 MTP layer).
• No training, fine-tuning or abliteration was performed here — this is a weight-only quantization.
  The behavioural characteristics, abliteration and usage warnings of the upstream model are
  inherited unchanged.
• Calibration data is general (math/code/logic/science QA); for heavy RP/creative use, a
  domain-matched calibration set may further improve fidelity.

License

Inherited from the upstream model, which declares the Apache License 2.0. See LICENSE and
NOTICE in this repository. Upstream chain: Qwen/Qwen3.8-27B → this checkpoint's base by DavidAU.