base_model: llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic
base_model_relation: quantized
quantized_by: DBMe
library_name: exllamav3
pipeline_tag: text-generation
license: apache-2.0
tags:
- exl3
- exllamav3
- quantized
- text-generation
gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic-exl3-mul1
EXL3 (ExLlamaV3) quantizations of llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic. All credit for the original model goes to the original authors.
📊 Available Quantizations & VRAM
The model weights are stored in separate branches. Please switch to a branch to download.
Note: VRAM estimates include PyTorch context overhead (~0.8GB) and assume an unquantized FP16 KV cache.
| Target BPW | Head BPW | Branch (Download Link) | WikiText-2 PPL (512 ctx)¹ | 2K ctx | 4K ctx | 8K ctx | 16K ctx | 32K ctx |
|---|---|---|---|---|---|---|---|---|
| 3.5 | h6 | 3.5bpw_h6 | 2335.9614 | N/A | N/A | N/A | N/A | N/A |
| 3.75 | h6 | 3.75bpw_h6 | 2312.4667 | N/A | N/A | N/A | N/A | N/A |
| 4.0 | h6 | 4.0bpw_h6 | 2097.6864 | N/A | N/A | N/A | N/A | N/A |
| 5.0 | h6 | 5.0bpw_h6 | 1995.7812 | N/A | N/A | N/A | N/A | N/A |
| 6.0 | h6 | 6.0bpw_h6 | 1995.3135 | N/A | N/A | N/A | N/A | N/A |
| 8.0 | h8 | 8.0bpw_h8 | 1985.1562 | N/A | N/A | N/A | N/A | N/A |
¹ Evaluated against WikiText-2 with ExLlamaV3 using a strided 512-token context window (-c 512) in llama.cpp parity mode (-g). Lower is better.
(Higher BPW = higher quality, lower BPW = fits in less VRAM).
📥 How to Download
It's recommended to use the huggingface-cli to download specific branches. (Do not use git clone as it will download all branches!)
Ensure you have the CLI installed:
pip install -U "huggingface_hub[cli]"
Download a specific branch (e.g., 5.0bpw_h6):
# Example: Downloading the 5.0bpw_h6 branch
huggingface-cli download DBMe/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic-exl3-mul1 --revision 5.0bpw_h6 --local-dir gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic-exl3-mul1-5.0bpw_h6
💻 Supported Engines
These models are highly optimized for modern GPUs and can be run using:
- TabbyAPI: A fast, OpenAI-compatible API server. (Set
model_nameto the local folder name you downloaded the branch into) - Text-Generation-WebUI: A local web interface. (Select the
exllamav3loader) - ExLlamaV3 (Native): Python library for custom integration.
📈 Perplexity Degradation Curve
(Lower is better)
⚙️ Advanced: Quantization Environment & Settings
🔬 Quantization Settings
Codebook: mul1
Output Scales: always
Calibration Rows: 250
Calibration Cols: 2048
Calibration Dataset: ExLlamaV3 Default (Wiki/C4/Code)
High Quality (HQ) Mode: False
ExLlamaV3:
1.0.0(Commit:cb7f2e9)Hardware:
NVIDIA RTX PRO 6000 Blackwell Server Edition