base_model: llmfan46/gemma-4-12B-it-qat-q4_0-unquantized-uncensored-heretic
base_model_relation: quantized
quantized_by: DBMe
library_name: exllamav3
pipeline_tag: text-generation
license: apache-2.0
tags:

• exl3
• exllamav3
• quantized
• text-generation

gemma-4-12B-it-qat-q4_0-unquantized-uncensored-heretic-exl3-mul1

EXL3 (ExLlamaV3) quantizations of llmfan46/gemma-4-12B-it-qat-q4_0-unquantized-uncensored-heretic. All credit for the original model goes to the original authors.

📊 Available Quantizations & VRAM

The model weights are stored in separate branches. Please switch to a branch to download.
Note: VRAM estimates include PyTorch context overhead (~0.8GB) and assume an unquantized FP16 KV cache.

¹ Evaluated against WikiText-2 with ExLlamaV3 using a strided 512-token context window (-c 512) in llama.cpp parity mode (-g). Lower is better.

(Higher BPW = higher quality, lower BPW = fits in less VRAM).

📥 How to Download

It's recommended to use the huggingface-cli to download specific branches. (Do not use git clone as it will download all branches!)

Ensure you have the CLI installed:

pip install -U "huggingface_hub[cli]"

Download a specific branch (e.g., 5.0bpw_h6):

# Example: Downloading the 5.0bpw_h6 branch
huggingface-cli download DBMe/gemma-4-12B-it-qat-q4_0-unquantized-uncensored-heretic-exl3-mul1 --revision 5.0bpw_h6 --local-dir gemma-4-12B-it-qat-q4_0-unquantized-uncensored-heretic-exl3-mul1-5.0bpw_h6

💻 Supported Engines

These models are highly optimized for modern GPUs and can be run using:

• TabbyAPI: A fast, OpenAI-compatible API server. (Set model_name to the local folder name you downloaded the branch into)
• Text-Generation-WebUI: A local web interface. (Select the exllamav3 loader)
• ExLlamaV3 (Native): Python library for custom integration.

📈 Perplexity Degradation Curve

(Lower is better)