license: apache-2.0
language:
- en
pipeline_tag: image-text-to-text
library_name: transformers
tags: - heretic
- qwen3_8
- qwen3_6
- uncensored
- finetune
- Cold Fusion
- GAIN Training
- Multi-stage tuning
- all use cases
- unsloth
- auto-round
- w4a16
- gptq
base_model: - Qwen/Qwen3.8-27B
- DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU (W4A16)
W4A16 GPTQ quantization of Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU using AutoRound with GPTQ group size 128.
Quantization Details
| Parameter | Value |
|---|---|
| Algorithm | AutoRound + GPTQ |
| Weight bits | 4 |
| Activation bits | 16 |
| Group size | 128 |
| Version | AutoRound 0.12.3 |
Model size: ~19GB (vs ~54GB FP16) — 65% reduction
Benchmark Comparison: W4A16 vs FP16
Both versions benchmarked on identical hardware (NVIDIA A100 80GB) using vLLM 0.27.1.
| Benchmark | FP16 | W4A16 | Delta |
|---|---|---|---|
| ARC-Easy (acc) | 86.91% | 87.25% | +0.34% |
| ARC-Easy (acc_norm) | 85.69% | 86.49% | +0.80% |
| HumanEval (pass@1) | 79.27% | 79.88% | +0.61% |
| WikiText2 PPL | 8.04 | 8.17 | +0.13 |
Key findings:
- W4A16 matches or exceeds FP16 on all classification/generation benchmarks
- Perplexity difference is negligible (0.13%) — well within measurement variance
- No perceivable quality loss at 4-bit weights with 16-bit activations
Usage
Works with any framework supporting GPTQ/AutoRound checkpoints:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-W4A16/DavidAU_Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-w4g128"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
torch_dtype="auto"
)
Or via vLLM for high-throughput serving:
vllm serve JC1DA/Qwen3.8-27B-DavidAU-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-INT4-W4A16 \
--tensor-parallel-size 2 \
--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \
--gpu-memory-utilization 0.85 \
--served-model-name Qwen3.8-27B-W4A16 \
--host 0.0.0.0 --port 8000 \
--enable-prefix-caching --kv-cache-dtype fp8
Model Card
See the FP16 original model card for full details on training methodology, stages, and capabilities.
This quantized version preserves all characteristics of the original:
- Heretic/uncensored output
- Strong reasoning and instruction following
- Reduced overthinking tokens
- Auto-variable thinking sizes