license: apache-2.0
base_model: DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
base_model_relation: quantized
library_name: transformers
pipeline_tag: image-text-to-text
tags:

• qwen3_5
• autoround
• w8a16
• int8
• 8-bit
• compressed-tensors
• vllm
• marlin
• mtp
• vision
• uncensored

Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU — W8A16 AutoRound

8-bit weight-only quantization (W8A16, int8 symmetric, group_size 128) of
DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
using AutoRound via llm-compressor 0.14

• auto-round 0.15.1, with 400 AutoRound iterations and 256 calibration samples.

• Quantized: all language-model Linear layers (400 modules: attention, MLP and GatedDeltaNet
  projections) → compressed-tensors pack-quantized, served by vLLM's Marlin kernels
  (uint8b128).
• Preserved in BF16 (untouched, bit-identical to the base model):
  • Vision tower (model.visual.*, 333 tensors)
  • MTP / Multi-Token Prediction head (mtp.*, 15 tensors) — enables speculative decoding
  • lm_head, embeddings, norms, conv1d, and the GatedDeltaNet in_proj_a/in_proj_b projections
    (output dim 48 < group_size 128)
• Calibration: 256 samples × 2048 tokens from
  neuralmagic/LLM_compression_calibration
  with the model's own chat template.
• Size: ~31.6 GB (base BF16: ~52 GB).

Quality

Perplexity measured on 50 held-out texts from wikitext-103, 16,007 evaluated tokens.
Deliberately not the calibration set. Same texts, same tokenization, same method (raw text, no
chat template), both models served by vLLM.

This checkpoint is the third and highest-fidelity arm of a series run on the same protocol. The
BF16 baseline reproduced identically (8.0880) across the W6A16 and W8A16 runs, so the deltas
below are directly comparable:

Reading this result

+5.91% is still above the ~5% degradation usually accepted for weight-only quantization, even
though this is 8-bit with double the tuning budget. Two honest caveats:

1. Three variables changed at once versus the W4A16 baseline — bit width, iterations and
   calibration samples. So this is a best-effort fidelity run, not a controlled isolation of
   the bit-width effect. Do not read the delta as "what 8 bits buys you".

2. The returns are clearly diminishing: +8.99% → +7.36% → +5.91%. Doubling bits and the
   tuning budget bought roughly one percentage point. That pattern suggests the residual error is
   not dominated by weight precision — if it were, 8-bit (int8, group 128) would typically land
   well under 1–2%.

A plausible next lever, and the one we would try first, is excluding the GatedDeltaNet
(linear-attention) projections from quantization — keeping in_proj_qkv, in_proj_z and the
linear-attention out_proj in BF16 — since hybrid linear-attention layers are a known sensitivity
in this architecture family. That trades some compression for fidelity instead of pushing the bit
width further on the same modules. Note this is a hypothesis, not a measured result.

Perplexity measures language-modelling fidelity only; it is not a substitute for task-specific
evaluation. Short side-by-side chat samples against the BF16 original read as equivalent, which is
exactly why the perplexity number — not eyeballing — is what is published here.

Serving notes

8-bit compressed-tensors weights are served by Marlin, so unlike the 6-bit build of this model
there is no dependency on the JIT-compiled Humming kernels:

Using MarlinLinearKernel for CompressedTensorsWNA16

No throughput advantage was measured. On the final checkpoints, same script and TP2, the W8A16
model was slower on plain decode (24.6 vs 26.5 tok/s) and faster with MTP speculative decoding
(42.6 vs 34.5) and on vision (19.4 vs 17.3). These are 2-prompt cold-JIT runs, not benchmarks; the
sign flips between phases, so no reliable throughput claim can be made in either direction. The
reason to prefer this build is fidelity, and it costs ~6 GB more than the 6-bit build.

Usage (vLLM ≥ 0.28)

vllm serve DoktorMincs/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W8A16-AutoRound \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --reasoning-parser qwen3 \
  --enable-prefix-caching \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

• Vision inputs work normally (image/video); the vision tower runs in BF16.
• The MTP draft head loads from the same checkpoint for speculative decoding.
• Validated on vLLM 0.28: text, MTP speculative decoding and vision all confirmed, 400 packed
  modules, in_proj_a/b correctly left in BF16.
• This is a thinking model: with a small max_tokens the whole budget can be consumed by the
  reasoning trace, leaving the visible answer empty. Use max_tokens ≥ 1024 and prefer the chat
  endpoint over raw completion.

Notes

• Architecture: Qwen3_5ForConditionalGeneration (hybrid: 64 layers, 3:1 GatedDeltaNet
  linear-attention : full-attention, 27-block ViT, 1 MTP layer).
• No training, fine-tuning or abliteration was performed here — this is a weight-only quantization.
  The behavioural characteristics, abliteration and usage warnings of the upstream model are
  inherited unchanged.
• Calibration data is general (math/code/logic/science QA); for heavy RP/creative use, a
  domain-matched calibration set may further improve fidelity.

License

Inherited from the upstream model, which declares the Apache License 2.0. See LICENSE and
NOTICE in this repository. Upstream chain: Qwen/Qwen3.8-27B → this checkpoint's base by DavidAU.