library_name: transformers
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/LICENSE
pipeline_tag: image-text-to-text
base_model:
- DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF
tags: - unsloth
- qwen
- qwen3_5
- GGUF
- llama.cpp
- MTP
datasets: - tristandruyen/calibration_data_v5_rc
metrics: - perplexity
Qwen3.6-27B-MTP-48Gb
Overview
This repository hosts a production-ready Q8_0 quantization of the exceptional Qwen3.6-27B fine-tune made by DavidAU and Nightmedia, specifically optimized to run within 48 GB of VRAM (dual RTX 3090s via NVLink). The weights were calibrated using the great iMatrix file calibration_data_v5_rc to precisely target layer-wise quantization sensitivity. On the tested hardware, this configuration achieves up to ~2,000 t/s prefill and ~65–80 t/s decode throughput with Multi-Token Prediction (MTP) enabled. The Q8_0 format was selected for its optimal speed-to-accuracy ratio, lower-precision variants (e.g., Q6_K) showed a 5 to 10% throughput drop alongside reduced long-context stability, capping reliable usage at ~242k tokens. By preserving critical precision in attention and KV cache operations, this quantization enables stable inference at the model’s maximum context window.
Note: Due to VRAM constraints, the MMPROJ vision encoder must be offloaded to CPU when running multimodal tasks.
Research & Methodology
Selective precision Quantization for high-speed inference
As previously disclosed, Q8_0 is the proper format but the model became too big for the 48Gb target. Therefore, the strategy was to use Q6_K on just a few tensors. I managed to identify which were the least sensitive to quantization thanks to the iMatrix, which also identified the few tensors that needed more care in order to maintain the model's capacities.
The schema identified was :
| Tensor | Layers | Format |
|---|---|---|
ssm_alpha |
57, 58, 60, 61, 62 | F16 |
ssm_beta |
57, 58, 60, 61, 62 | F16 |
attn_q, attn_k, attn_v |
3, 7 | F16 |
attn_output |
3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47, 51, 55 | Q6_K |
ssm_out |
0–2, 4–6, 8–10, 12–14, 16–18, 20–22, 24–26, 28–30, 32–34, 36–38, 40–42, 44–46, 48–50, 52–54, 56–58 | Q6_K |
Therefore, 93.91% of the model is Q8_0, 5.44% is Q6_K and 0.61% is F16.
iMatrix Calibration
The model was calibrated using the very well, and rightfully, known calibration data calibration_data_v5_rc. I tested several iMatrix dataset, including of my own making, but this file achieved the best results.
Accuracy Analysis
To evaluate the precision loss after the mixed-quantization scheme, I compared the perplexity on Wiki-Text-raw and a custom made dataset, composed of code mainly found on llama.cpp's repo:
| Metric | Code | WikiText-Raw |
|---|---|---|
| Base Mean Perplexity | 1.879303 ± 0.005783 | 6.320168 ± 0.039392 |
| Quant Mean Perplexity | 1.879671 ± 0.005789 | 6.324147 ± 0.039467 |
| PPL(Q) / PPL(Base) | 1.000196 ± 0.000094 | 1.000630 ± 0.000145 |
| Cor(ln PPL) | 99.95% | 99.97% |
| Mean KLD | 0.000795 ± 0.000069 | 0.000746 ± 0.000023 |
| 99.9% KLD | 0.032576 | 0.031017 |
| 99.0% KLD | 0.008418 | 0.006217 |
| 95.0% KLD | 0.002935 | 0.002095 |
| Median KLD | 0.000029 | 0.000304 |
| Max KLD | 12.675076 | 1.559480 |
| RMS Δp | 0.885 ± 0.030 % | 0.773 ± 0.031 % |
| Same Top-p | N/A | 98.806 ± 0.028 % |
Key Findings:
- Near-Lossless: The perplexity degradation is minimal at +0.003979 on Wiki-Text-Raw, indicating that this mixed precision layout preserves the original model's reasoning and coding capabilities.
- Reaches full-Q8_0 capacities: Other tests made on a 'pure' Q8_0 quantization shown the same perplexity and KL-Div results, proof that this work has achieved its goals, lower memory footprint with no accuracy loss.
Recommended Usage
To replicate the optimal performance (262K context, F16 K-Cache, Multi-GPU) using llama.cpp, use the following llama-server command.
Note the specific use of --split-mode tensor and --tensor-split 1,1 for optimal PCIe bandwidth management across dual RTX 3090s. This command appeared to be the best one I could come across using an NVLink.
As previously disclosed, the mmproj file will not hold on the GPU, if you want to use it you need to add --no-mmproj-offload and --image-min-tokens 1024 to the following command, alongside the --mmproj /path/to/mmproj-BF16.gguf.
/path/to/llama.cpp/build/bin/llama-server \
-m /path/to/Qwen3.6-27B-Fusion-711-Uncensored-MTP-Q8_0-Opt.gguf \
--split-mode tensor \
--tensor-split 1,1 \
--host 0.0.0.0 \
--port 8080 \
--ctx-size 262144 \
--parallel 1 \
--gpu-layers 999 \
--cache-type-k f16 \
--cache-type-v q8_0 \
--flash-attn on \
-b 2048 -ub 2048 \
--spec-type draft-mtp \
--alias Qwen3.6-27b \
--jinja
I also developed a proxy to enable users to select thinking or non-thinking behaviors, apply the recommended sampling parameters AND the "Preserve Thinking" option. You may find it on my GitHub.
Hardware Requirements
- Target VRAM: 48 GB (Tested on 2x NVIDIA RTX 3090 24GB).
- RAM: Minimum 32GB system RAM (Prompt caching, system overhead and MMPROJ).
- Context limit: The command above loads ~21GB of KV cache across the two GPUs. If you experience OOM (Out of Memory) errors, consider reducing
--ctx-sizeor using 8-bit K-cache (--cache-type-k q8_0).
Acknowledgments
This project was made possible thanks to the outstanding tools and contributions from the open-source AI community. Special thanks to:
- DavidAU and Nightmedia: For their immense work done on this exceptionnal fine-tuned version.
- The Qwen Team: For researching and releasing the exceptional Qwen3.6 architecture.
- llama.cpp: Using the new Tensor split mode, it finally achieves extremelly high performances on dual-GPU setups.
- Froggeric: For the immense work done on fixing Qwen3.5 and 3.6 chat_template and the base BF16 model used for this project.
- tristandruyen: For the great calibration_data_v5-rc.txt file.