license: apache-2.0
base_model: llmfan46/gemma-4-31B-it-uncensored-heretic
base_model_relation: quantized
tags:
- exl3
- exllamav3
- gemma4
- heretic
- uncensored
- vision
inference: false
gemma-4-31B-it-uncensored-heretic EXL3 3.00bpw h4
EXL3 export of llmfan46/gemma-4-31B-it-uncensored-heretic
for ExLlamaV3 / TabbyAPI. Vision tower included, quantized to 6 bpw.
Load with ExLlamaV3 or TabbyAPI's exllamav3 loader. This will not load in
Transformers, vLLM or llama.cpp.
This is the 4-bit-head companion to-h6.
Same 3.0 bpw weights; the head and the vision tower are smaller. See
Differences from h6.
Quantization
From quantization_config.json:
| Method | EXL3 |
| ExLlamaV3 | 1.5.0 |
| Weights | 3.0 bpw |
| Head | 4 bits |
| Vision | 6 bpw |
| Codebook | mul1 |
| Output scales | always |
| Calibration | 250 rows x 2048 cols |
830 language-model modules at 3 bpw, lm_head at 4 bits, 354 vision modules at
6 bpw. embed_tokens is carried unchanged in BF16 (2.625 GiB) — ExLlamaV3 keeps
it in system RAM rather than VRAM.
config.json carries an audio_config, but the checkpoint has no audio tensors
— neither does the source or google/gemma-4-31B-it. This is an image+text model.
Converted on an RTX 4090 on 2026-09-19 from the BF16 source (62.5 GB).
Differences from h6
| h6 | this (h4) | |
|---|---|---|
| ExLlamaV3 | 1.4.2 | 1.5.0 |
lm_head |
6 bits, 0.985 GiB | 4 bits, 0.657 GiB |
| Vision tower | BF16, 1.061 GiB | 6 bpw, 0.430 GiB |
| Total | 14.927 GiB | 13.961 GiB |
The language-model layers are the same size in both, 10.245 GiB — same 3.0 bpw
over the same shapes. Only the head and the vision tower changed size.
The vision difference is not a choice: ExLlamaV3 1.4.2 could not quantize Gemma 4's
vision tower at all, so h6 stores it at BF16. 1.5.0 can.
Size
| File | Bytes |
|---|---|
model-00001-of-00002.safetensors |
8,498,844,777 |
model-00002-of-00002.safetensors |
6,492,009,964 |
| Total | 14,990,854,741 (13.961 GiB) |
Component breakdown, summed from the safetensors headers:
| GiB | |
|---|---|
| Language-model layers | 10.245 |
embed_tokens (BF16, host RAM) |
2.625 |
lm_head |
0.657 |
| Vision tower | 0.430 |
Measured VRAM
Vision disabled, cache_mode: 3,3, real prefills at ~95% of the window — not
just successful loads.
RTX 4090, 24 GB (chunk_size: 128, max_batch_size: 2):
| max_seq_len | prefill peak |
|---|---|
| 98,304 | 17,435 MiB |
| 131,072 | 17,995 MiB |
| 163,840 | 19,573 MiB |
| 196,608 | 20,133 MiB |
| 262,144 | 21,253 MiB |
262,144 was the top of the sweep, not a limit — 3.3 GB was still free.
RTX 5080, 16 GB (chunk_size: 256, max_batch_size: 1):
| max_seq_len | prefill peak |
|---|---|
| 61,440 | 15,021 MiB |
| 73,728 | 15,741 MiB |
| 81,920 | 15,837 MiB |
| 90,112 | fails mid-prefill |
These are single-prefill verifications. A window that survives one clean prefill
can still OOM under sustained traffic, so run below the number in the table.
Only 10 of Gemma 4's 60 layers hold a per-token K/V cache — the other 50 are
sliding-window with fixed-size state — so at 3,3 context costs about
17.9 KiB/token, far less than the layer count suggests. On a card near its limit
the prefill working set, not the cache, is what binds.
Vision
The tower costs 454 MiB of VRAM. Setting EXL3_VISION_PINNED=1 holds it in
pinned host memory instead, cutting that to 54 MiB (measured at 61,440 on a
5080). Nothing else offloads — that flag applies to the vision component only.
Serving
TabbyAPI, as run here on a 4090:
max_seq_len: 98304
cache_size: 98304
cache_mode: 3,3
chunk_size: 128
gpu_split_auto: true
prompt_template: gemma4
chat_template.jinja is a separate file — Gemma 4 does not put the template intokenizer_config.json.
Quality
Not evaluated. The numbers above are memory and context measurements; no
benchmark or side-by-side against h6 has been run. 3.0 bpw with a 4-bit head is
an aggressive quant chosen to fit long context on one card, and a 4-bit head is
a further reduction on top of an already lossy export. If you have the VRAM,
prefer a higher-bpw build.
Credit and license
Apache 2.0, matching google/gemma-4-31B-it and the source model.
Decensoring credit belongs to llmfan46 — this
repo is only the EXL3 export. Quantization by
ExLlamaV3.