language:
- en
license: apache-2.0
tags: - qwen3.6
- gptq
- int4
- quantized
- dynamic-quantization
- mtp
- multi-token-prediction
- uncensored
- reasoning
- consumer-hardware
pipeline_tag: text-generation
base_model: - DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP
library_name: gptqmodel
inference_library_version: "7.3.2"
quantization_config:
quant_method: gptq
bits: 4
group_size: 128
desc_act: true
Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-GPTQ-INT4
GPTQ INT4 quantization of the original DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP, optimized for dual-RTX-2080-Ti deployment.
Quantization Details
- Method: GPTQ (via modelcloud/gptqmodel v7.3.2)
- Base precision: INT4 (4-bit weights, group_size=128, symmetric, desc_act=true)
- Dynamic per-layer override: Layers 68-79 projection layers quantized at INT8 instead of INT4 to preserve reasoning quality in the upper transformer layers
- LM head: Not quantized (preserved at BF16)
- Total size: ~19 GB (vs ~56 GB BF16 original)
Deployment Configuration
This quantization was specifically optimized and tested on a dual-RTX-2080-Ti (22GB each) setup with NVLink interconnect, running a custom vLLM fork designed for consumer GPUs.
- Inference engine: weicj/vLLM-2080Ti-Definitive v0.1.14 (fork of vLLM with SM75-specific patches and INT4 GPTQ kernel optimizations)
- GPU config: 2× NVIDIA RTX 2080 Ti (22GB VRAM each, compute capability 7.5), connected via NVLink (2 links per GPU, 25.781 GB/s each)
- Tensor parallelism: TP=2
- Context length: 262K tokens
- MAX_NUM_SEQS: 2
- GPU memory utilization: 0.93
- MTP depth: 3
Benchmark Results
Measured on dual RTX 2080 Ti (NVLink) with the configuration above. All tests generate 1024 output tokens.
| Prompt Length (tokens) | TTFT (ms) | ITL Avg (ms) | ITL Std (ms) | Prefill Time (ms) | Prefill Speed (tok/s) | Output Time (ms) | Decode Speed (tok/s) |
|---|---|---|---|---|---|---|---|
| 4,096 | 3,265 | 39.99 | 0.50 | 3,260 | 1,256 | 15,358 | 66.68 |
| 8,192 | 6,730 | 40.62 | 0.79 | 6,725 | 1,218 | 11,903 | 86.03 |
| 16,384 | 14,034 | 41.46 | 0.64 | 14,029 | 1,168 | 17,537 | 58.39 |
| 32,768 | 29,943 | 42.84 | 1.10 | 29,937 | 1,095 | 16,194 | 63.23 |
| 65,536 | 66,835 | 45.29 | 1.25 | 66,829 | 981 | 16,531 | 61.94 |
| 131,072 | 160,941 | 50.18 | 2.40 | 160,936 | 814 | 19,620 | 52.19 |
- TTFT: Time To First Token
- ITL: Inter-Token Latency (average between consecutive output tokens)
- Prefill Speed: prompt tokens processed per second
- Decode Speed: output tokens generated per second
Usage
Load with vLLM:
from vllm import LLMllm = LLM(
model="cjxzdzh/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-GPTQ-INT4",
quantize="gptq",
tensor_parallel_size=2,
max_model_len=262144,
gpu_memory_utilization=0.93,
max_num_seqs=2,
)
Load with GPTQModel directly:
from gptqmodel import GPTQModel, QuantizeConfigmodel = GPTQModel.from_quantized(
"cjxzdzh/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-GPTQ-INT4",
device_map="auto",
)
Notes
- The dynamic INT8 override on layers 68-79 projection layers is intentional and required for maintaining reasoning quality on this model family. This is a per-layer selective quantization strategy, not a uniform INT8 quantization.
- The forked vLLM (weicj/vLLM-2080Ti-Definitive) includes critical patches for SM75 architecture and INT4 GPTQ kernel optimizations that are not present in upstream vLLM.
- For benchmark results on the base model, refer to the original repository.