license: apache-2.0
base_model:
- DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: transformers
tags: - auto-round
- int4
- w4a16
- gptq
- mtp
- intel
- xpu
- arc-pro-b70
- vllm
- heretic
- uncensored
- abliterated
Qwen3.8-27B TURBO Fable Cold-Fusion 735-882 Heretic NM-DAU: int4 AutoRound (MTP int4)
int4 weight-only (W4A16) AutoRound quantization of
DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
@ 079b2b268c637b77f2fb4ab6a5ca1edc47d462a5, made with the recipe of
Frozenlock/Qwen3.8-27B-int4-AutoRound
@ b4c61732. The MTP head is quantized too, which leaves more memory for the
KV cache under multi-token-prediction speculative decoding (197,982 tokens at
131k context on one Intel Arc Pro B70). Tested on the B70 with vLLM-XPU.
Uncensored model. The source checkpoint is "Heretic"-abliterated: its
author removed the refusal behaviour, so it answers requests that base Qwen3.8
declines. This quant keeps that behaviour. Add your own safeguards if you serve
it to other people.
Quantization
| field | value |
|---|---|
| tool | auto-round 0.14.2 (torch 2.12.0+xpu, transformers 5.8.0) |
| scheme | W4A16, int, symmetric, group size 128 |
| tuning | AutoRound defaults: iters 200, nsamples 128, seqlen 2048, NeelNanda/pile-10k; batch 4 x grad-accum 2, low_gpu_mem_usage |
| packing | auto_round:auto_gptq (quant_method: auto-round) |
| kept bf16 | GDN in_proj_a / in_proj_b (48 layers each), mtp.fc, vision tower, embeddings, lm_head |
| MTP layers | int4, RTN from the source weights (auto-round's missing_tensors path, as in the recipe reference) |
| hardware | Intel Arc Pro B70, 302 min of tuning |
| size | 19.0 GB, 27.8B parameters (the Hub's "6B params" badge counts packed I32 storage: 8 int4 weights per entry) |
MANIFEST.json records the pins and tensor counts. recipe/bake-autoround.sh
is the exact script that produced these weights. It refuses any other source
revision, and before saving it checks the produced quantization_config
against the reference.
Results on an Intel Arc Pro B70
Image: intel/llm-scaler-vllm:0.26.0-b2 plus local patches (see "Serving"),--quantization inc, --dtype float16, MTP speculative decoding with 3
tokens over a 32k-token draft vocabulary shortlist, fp8 KV cache, 131,072
context, prefix caching, max 16 sequences. Compared with the other auto_round:auto_gptq
int4 of the same source,
JC1DA/…-INT4-W4A16
@ 3b19cdc9 (iters 512, seqlen 4096, MTP kept bf16), on the same stack on the
same day. Both quants come from the same source weights (sha256 of 9 tensors
both keep bf16 match).
| this quant | JC1DA | |
|---|---|---|
| KV cache pool | 197,982 tokens | 183,317 tokens |
| per-request tok/s, 1 / 4 / 16 concurrent (300-token answers) | 72.2 / 55.5 / 33.8 | 67.5 / 54.4 / 33.3 |
| time to first token, 4k / 32k / 64k prompt | 1.72 / 18.0 / 46.7 s | 1.77 / 18.2 / 50.2 s |
| MTP acceptance (cumulative) | 0.70 | 0.72 |
| internal 50-task eval (JSON, tools, code, reasoning), default effort | 50/50 | 50/50, twice |
internal 50-task eval, reasoning_effort: low |
49/50, then 50/50 | 49/50 |
| NaN-logit and degeneration probes, 100k needle, 6 concurrent JSON-mode requests | pass | pass |
| 90 min sustained-load soak | 750 requests, 0 errors | not run |
Quality is a tie. The speed difference is small but shows on every cell, most
likely because the int4 MTP head is lighter than a bf16 one (not isolated).
The internal eval is a regression check, not a benchmark; run your own evals
for anything that matters.
Serving
vLLM, any loader that reads auto_round:auto_gptq int4. The numbers above
come from intel/llm-scaler-vllm:0.26.0-b2 with patches applied: upstream
vLLM PR 50501 (W4A8 prefill), llm-scaler PR 700, and two local patches, one
that runs inc AutoRound int4 on llm-scaler's sym_int4 kernels and one that
restricts the MTP drafter to a 32k-token vocabulary shortlist. A stock image is
untested with this checkpoint.
The core flags we serve with (bf16 activations fail on the B70, so--dtype float16):
vllm serve <this repo> --quantization inc --dtype float16 --max-model-len 131072 \
--max-num-seqs 16 --block-size 64 --enable-prefix-caching \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--kv-cache-dtype fp8 --mamba-ssm-cache-dtype float16 \
--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
The chat template and generation config are the source's: temperature 1.0,
top_k 20, top_p 0.95; the template takes reasoning_effort xhigh (default),
medium or low.
Credits
All model quality is DavidAU's fine-tune of Qwen/Qwen3.8-27B; see the
source model card.
The recipe is Frozenlock's. Apache-2.0, as the source.