license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- qwen3_5
- quantized
- w4a16
- auto-round
- compressed-tensors
- vllm
- uncensored
base_model: DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W4A16-AutoRound
W4A16 pack-quantized (compressed-tensors, vLLM-ready) version of
DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
— a Qwen3.8-27B (qwen3_5) finetune, Heretic/uncensored.
Built for single-GPU (RTX 3090 24 GB) vLLM serving with long context: the main
transformer body is int4, lm_head / embeddings / MTP draft module are int8, and a
vocab-truncated draft head (40960 tokens) is included for MTP speculative decoding.
Note on use: the base model is uncensored (refusal behavior removed by its
authors). Responsibility for how you use this quantization lies with you.
What this is
| Base model | DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU (BF16, ~55 GB) |
| Architecture | Qwen3.5 (qwen3_5), 27B, 64 layers (48 linear-attention DeltaNet + 16 full attention), vocab 248320, MTP module, vision tower, untied embeddings |
| Quantization | W4A16 AutoRound (int4, group 128, symmetric) + int8 lm_head / embed_tokens / MTP |
| Format | compressed-tensors pack-quantized (weight_packed int32 + weight_scale + weight_shape) |
| Size on disk | ~16 GB |
| Tested with | vLLM 0.27.1, transformers 5.15.0, CUDA 13, RTX 3090 24 GB |
How much context fits (measured on RTX 3090)
| Config | KV pool | Result |
|---|---|---|
gpu_memory_utilization=0.94, max_model_len=140000 (recommended) |
152,173 tokens | fits with ~9% headroom |
gpu_memory_utilization=0.96, max_model_len=160000 |
167,692 tokens | fits, less runtime headroom |
gpu_memory_utilization=0.93, max_model_len=160000 |
~146k ceiling | does not fit (engine refuses to start) |
Pool size varies a few percent between boots; 140k @ 0.94 is the safe everyday setting.
FP8 KV cache (--kv-cache-dtype fp8), prefix caching on, MTP speculative decoding with 3 draft tokens.
Run it (example)
vllm serve ababaka/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W4A16-AutoRound \
--served-model-name qwen3.8-27b \
--host 0.0.0.0 --port 18020 \
--gpu-memory-utilization 0.94 \
--max-model-len 140000 \
--max-num-seqs 8 \
--language-model-only \
--kv-cache-dtype fp8 \
--mamba-ssm-cache-dtype float16 \
--async-scheduling \
--max-num-batched-tokens 2048 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}' \
--compilation-config '{"max_cudagraph_capture_size":32,"custom_ops":["+rms_norm","+silu_and_mul"]}' \
--reasoning-parser qwen3 \
--enable-auto-tool-choice --tool-call-parser qwen3_xml \
--enable-prefix-caching --mamba-cache-mode align
For 160k context use --gpu-memory-utilization 0.96 --max-model-len 160000.--language-model-only skips the vision tower (text serving). The chat template of the
base model is used by default; a Sharp custom template
(peculiar-ragdoll/Qwen-Sharp-Chat-Templates, reasoning_effort via--default-chat-template-kwargs) was used in testing.
Files
model-00001..07-of-00007.safetensors— quantized transformer + int8lm_head/embed_tokensmodel_extra_tensors.safetensors— int8 MTP module +mtp.draft_lm_head(40960-token draft head)mtp_draft_vocab_ids.pt— token-id map for the truncated draft vocabularymodel.safetensors.index.json,config.json(+quantization_config.json), tokenizer/processor configs
How this quantization was made (recipe, pipeline, format)
Base
Downloaded DavidAU/...-Heretic-Uncensored-NM-DAU in FP16 (13 shards incl.model-mtp-restored.safetensors, ~55 GB). Verified same skeleton as the reference
Qwen3.8-27B quant: qwen3_5, 64 layers (48 linear + 16 full), vocab 248320,tie_word_embeddings=false, mtp.fc + 7 MTP linears, model.visual.* tower.
Quantization (auto-round 0.14.2)
| Parameter | Value |
|---|---|
| Toolchain | auto-round 0.14.2, transformers 5.15.0, torch 2.13+cu130, compressed-tensors |
| Scheme | W4A16, int4, group 128, symmetric (pack-quantized) |
| Dataset | NeelNanda/pile-10k, 128 samples, seqlen 2048, batch 4 |
| Iters | 200 |
| seed / trust_remote_code | 42 / True |
| quant_nontext_module | False (vision tower untouched) |
| Export format | llm_compressor |
| VRAM adaptation | low_gpu_mem_usage=True (CPU offload for 24 GB cards) |
layer_config kept in BF16: lm_head, linear_attn.in_proj_a/b (all 48 linear-attention
layers), visual.* (whole vision tower), mtp (whole MTP module).
Full run on RTX 3090: 64/64 layers in ~2h41m (~141–159 s/layer), peak VRAM 15.5 GB /
RAM 27.3 GB. Result: 400/607 submodules quantized; ignore list 303 entries
(110 visual + 192 linear_attn + lm_head), no mtp/embed entries — MTP was copied
unquantized into model_extra_tensors.safetensors by the exporter.
Post-quantization pipeline (in this order)
- quant_lm_head —
lm_head→ int8 g128 (round-trip rel. error 0.64%, ~1.3 GB VRAM saved) - quant_embed —
embed_tokens→ int8 g128, scales in bf16 (untied embeddings, error 0.60%, ~1.3 GB saved) - quant_mtp —
mtp.fc+ 7 MTP linears → int8 g128 (errors 0.6–1.5%) - build_draft_vocab — draft head sliced from the int8
lm_headto the 40960 most
frequent token ids →mtp.draft_lm_head.*(213 MB) inmodel_extra_tensors.safetensorsmtp_draft_vocab_ids.pt
Final config_groups (order matters for compressed-tensors matching in vLLM —
first match wins, so the lm_head regex must precede the mtp one):
group_0 targets=["Linear"] int4 g128 sym — main body
group_1 targets=["re:.*lm_head$"] int8 g128 — lm_head + mtp.draft_lm_head
group_2 targets=["re:.*embed_tokens$"] int8 g128 — embed_tokens
group_3 targets=["re:^mtp\\..*"] int8 g128 — MTP module
ignore: 302 entries (visual + linear_attn norms/projections); lm_head/mtp/embed
are NOT ignored. Verified by loading in vLLM: MTP drafter detected, 40960-token draft
head recognized, coherent generation (no random-token garbage).
License
Base model is Apache 2.0; this quantization follows the base model's license.