license: apache-2.0
base_model: llmfan46/gemma-4-31B-it-uncensored-heretic
base_model_relation: quantized
tags:
- exl3
- exllamav3
- gemma4
- heretic
- uncensored
- vision
inference: false
gemma-4-31B-it-uncensored-heretic EXL3 3.00bpw h6
EXL3 export of llmfan46/gemma-4-31B-it-uncensored-heretic
for ExLlamaV3 / TabbyAPI. Vision tower included.
Load with ExLlamaV3 or TabbyAPI's exllamav3 loader. This will not load in
Transformers, vLLM or llama.cpp.
Quantization
From quantization_config.json:
| Method | EXL3 |
| ExLlamaV3 | 1.4.2 |
| Weights | 3.0 bpw |
| Head | 6 bits |
| Codebook | mul1 |
| Output scales | always |
| Calibration | 250 rows x 2048 cols |
All 832 language-model tensors are quantized. All 356 vision tensors are carried
unchanged in BF16 in the same shards, so image input works without a second
download. config.json carries an audio_config, but the checkpoint has no
audio tensors — neither does the source or google/gemma-4-31B-it, so this is
an image+text model.
Converted on an RTX 4090 on 2026-09-10 from the BF16 source (62.5 GB).
Files
| File | Notes |
|---|---|
model-0000{1,2}-of-00002.safetensors |
EXL3 weights + BF16 vision, 16.06 GB total |
quantization_config.json |
EXL3 parameters, per-tensor storage map |
chat_template.jinja |
Gemma 4 chat template (it is not in tokenizer_config.json) |
processor_config.json |
Gemma 4 vision processor |
tokenizer.json / tokenizer_config.json |
tokenizer |
Serving
Run here daily on a 24 GB RTX 4090 under TabbyAPI:
max_seq_len: 98304
cache_size: 98304
cache_mode: 3,3 # Q3 K/V — needed to fit 96K context
chunk_size: 128
gpu_split_auto: true
prompt_template: gemma4
Measured 17.4 GiB VRAM with the vision tower loaded and the full 96K Q3 cache
allocated. Weights alone are 14.9 GiB, so a 16 GB card fits the weights but has
almost nothing left for context.
At 3.0 bpw this is a lossy quant chosen to fit long context on one 4090. If you
have the VRAM, a 4.0+ bpw export will be noticeably better.
Credit and license
Apache 2.0, matching google/gemma-4-31B-it and the source model.
Decensoring credit belongs to llmfan46 — this
repo is only the EXL3 export. Quantization by
ExLlamaV3.