license: apache-2.0
license_link: https://ai.google.dev/gemma/docs/gemma_4_license
library_name: gguf
base_model: abacusai/Gemma-4-12B-Uncensored
base_model_relation: quantized
pipeline_tag: image-text-to-text
tags:
- gguf
- llama.cpp
- gemma4
- uncensored
- abacusai
- smaug
- dpo
1. Introduction
GGUF builds of Gemma-4-12B-Uncensored for llama.cpp, Ollama and LM Studio: five sizes from IQ4_XS to Q8_0, plus the image projector and Google's assistant draft model for faster generation. The model is Gemma 4 12B-it with its refusals trained out by Abacus.AI. On 241 harmful prompts from HarmBench and JailbreakBench, stock Gemma 4 12B-it refuses 91.7% with thinking on; the Q4_K_M file here refuses 4.1% (10 of 241), measured on the file itself in llama.cpp and the same as the full-precision weights. With thinking off the figures are 95.4% and 16.6% (40 of 241). Every file was loaded and tested with thinking on, thinking off and an image, and the default download was run in llama.cpp, Ollama and LM Studio.
Built with llama.cpp b8abe41 from the BF16 weights. Q4_K_M and IQ4_XS use an importance matrix (imatrix.gguf, included) computed on 48K tokens of prose, code, dialogue, math, science and multilingual text. Training, the full refusal evaluation and the known limitations are on the source card.
Refusal rates
Share of 241 harmful prompts refused or hedged, scored by an LLM judge on the full answer. Lower means the model answered. The Q4_K_M bar is the file in this repository run in llama.cpp; the other two bars are full-precision weights.
2. Which file to pick
| Quant | File | GPU memory 8K context | GPU memory 32K context | Pick it when you have |
|---|---|---|---|---|
IQ4_XS | 6.64 GB | 7.9 GB | 8.3 GB | Smallest file. 10 GB of GPU memory; on an 8 GB card a few layers go to the CPU. |
Q4_K_M | 7.38 GB | 8.6 GB | 9.0 GB | The default (llama.cpp and Ollama pick it). 12 GB of GPU memory. |
Q5_K_M | 8.55 GB | 9.6 GB | 10.0 GB | 12 GB with a shorter context. |
Q6_K | 9.79 GB | 10.8 GB | 11.2 GB | 16 GB, or a 12 GB card with a short context. |
Q8_0 | 12.67 GB | 13.5 GB | 13.9 GB | Close to full precision. 16 GB and up. |
3. Quick start
llama.cpp (downloads the image projector too):
llama-server -hf abacusai/Gemma-4-12B-Uncensored-GGUF:Q4_K_M --jinja -ngl all -c 8192 \
--temp 1.0 --top-p 0.95 --top-k 64
The model thinks by default, and thinking on is the mode it was tuned for. To turn thinking off, add --chat-template-kwargs '{"enable_thinking": false}', or send "chat_template_kwargs": {"enable_thinking": false} with a request.
Faster generation with the draft model (llama.cpp): download the draft file and add it to the command.
hf download abacusai/Gemma-4-12B-Uncensored-GGUF mtp-gemma-4-12B-it-Q8_0.gguf --local-dir .
llama-server -hf abacusai/Gemma-4-12B-Uncensored-GGUF:Q4_K_M --jinja -ngl all -c 8192 \
--temp 1.0 --top-p 0.95 --top-k 64 \
-md mtp-gemma-4-12B-it-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 3 -ngld all
Ollama:
ollama run hf.co/abacusai/Gemma-4-12B-Uncensored-GGUF:Q4_K_M
Thinking is on by default and --think=false turns it off. Ollama 0.40.16 rejects an explicit "think": true for Gemma 4 GGUF files pulled from the Hub, so leave it unset to keep thinking on.
LM Studio: search for abacusai/Gemma-4-12B-Uncensored-GGUF and pick a file from the table above.
Recommended sampling is the base model's: temperature 1.0, top_p 0.95, top_k 64.
4. Checks on the quantized files
Refusals, Q4_K_M vs full precision
Quantising to Q4_K_M did not shift refusals with thinking on: the file and the full-precision weights each refuse 10 of the 241 harmful prompts (paired exact test, p 1.0). With thinking off the file refuses 40 against 31; 16 prompts are refused only by Q4_K_M and 7 only by the full-precision weights (p 0.09), a gap this prompt set cannot separate from sampling noise. Stock Gemma 4 12B-it refuses 221 and 230.
| Q4_K_M llama.cpp | Full precision vLLM | Gemma 4 12B-it base, full precision, vLLM | |
|---|---|---|---|
| Harmful prompts refused, thinking on the default; lower is better |
4.1 10/241 |
4.1 10/241 |
91.7 221/241 |
| Harmful prompts refused, thinking off lower is better |
16.6 40/241 |
12.9 31/241 |
95.4 230/241 |
Capability of each file
| BF16 full precision | Q8_0 | Q6_K | Q5_K_M | Q4_K_M | IQ4_XS | |
|---|---|---|---|---|---|---|
| MMLU-Pro subset 280 questions, exact match |
75.4 211/280 |
75.0 210/280 |
76.8 215/280 |
73.6 206/280 |
73.6 206/280 |
70.7 198/280 |
| IFEval 541 prompts, prompt-level loose |
95.6 517/541 |
95.7 518/541 |
95.7 518/541 |
95.4 516/541 |
95.0 514/541 |
93.3 505/541 |
| Answers cut off at the cap of 821; lower is better |
1.0 8/821 |
0.5 4/821 |
1.0 8/821 |
1.3 11/821 |
1.2 10/821 |
3.7 30/821 |
Numerical drift
These files track their own full-precision weights more closely than stock Gemma 4 12B-it quantized the same way tracks its own, at every size. The absolute values are far higher than for most model families because Gemma 4 handles this raw-text test poorly even in full precision, so read the table as a comparison with stock and use the task scores above to choose a file.
| Quant | Mean KLD this repository | Mean KLD stock Gemma 4 12B-it | Same top token this repository | Same top token stock Gemma 4 12B-it |
|---|---|---|---|---|
IQ4_XS | 0.886 | 1.209 | 69.5% | 59.8% |
Q4_K_M | 0.624 | 0.795 | 73.3% | 67.2% |
Q5_K_M | 0.669 | 0.972 | 73.6% | 64.0% |
Q6_K | 0.366 | 0.528 | 81.9% | 74.1% |
Q8_0 | 0.193 | 0.285 | 87.9% | 83.9% |
llama-perplexity --kl-divergence; lower is closer. "Same top token" is how often the quant's most likely next token equals BF16's. The stock columns are google/gemma-4-12B-it converted and quantized here with the same converter, settings and calibration text, each file against its own BF16. Full-precision perplexity on this text is 595 for stock Gemma 4 12B-it and 692 for this model. Stock builds by others, measured the same way against the stock BF16: the ggml-org Q8_0 gives exactly the stock Q8_0 values above, and the bartowski Q4_K_M 0.837 (66.0%). Bold = closer of the pair.Draft model
mtp-gemma-4-12B-it-Q8_0.gguf is Google's assistant draft model for Gemma 4 12B-it, converted unchanged; it was not retrained for this model and still drafts well for it. With Q4_K_M and 3 draft tokens, 61.9% of drafted tokens were accepted and generation ran at 228 tokens per second against 119 without the draft (1.91x), on 24 prompts (12 MMLU-Pro, 12 IFEval) with thinking on and up to 1,024 tokens each, one request at a time on an RTX PRO 6000.
Chat template
llama.cpp renders this repo's embedded chat template exactly as the Hugging Face tokenizer does (checked for a thinking-on turn, a thinking-off turn and a tool-call round trip), so the thinking switch and the tool-call format are the same as in the full-precision release.
Not covered
Refusals were measured on Q4_K_M only. Audio and video input, the full 262,144-token context, tool-call accuracy, macOS and Windows were not tested.
5. License
Apache 2.0, inherited from Gemma 4 12B-it. See LICENSE.
6. Citation
@misc{abacusai2026gemma4uncensored,
title = {Gemma-4-12B-Uncensored},
author = {Abacus.AI},
year = {2026},
note = {Refusal-removal fine-tune of google/gemma-4-12B-it by teacher-distilled DPO},
url = {https://huggingface.co/abacusai/Gemma-4-12B-Uncensored}
}
7. Contact Us
If you have any questions, please reach out at Abacus.AI.