license: apache-2.0
base_model: TrevorJS/gemma-4-12B-it-uncensored
library_name: gguf
tags:
- gguf
- gemma
- gemma4
- uncensored
- kleidiai
- llama.cpp
Gemma 4 12B IT — Uncensored (KleidiAI-optimized quants)
Uncensored fine-tune of Gemma 4 12B IT,
quantized by hand for llama.cpp targeting KleidiAI-optimized ARM kernels
(Q4_0 / Q8_0). 48 layers, 262K context.
Files
| File | Size | Purpose |
|---|---|---|
gemma-4-12B-it-uncensored-KleidiAI.gguf |
6.6 GB | Main model. 667 tensors: 274x Q4_0 (24.3B elems), 55x Q8_0 (0.55B elems), 338x F32 (norms/scales) |
mmproj-gemma-4-12B-it-bf16.gguf |
168 MB | Multimodal projector (bf16), vision (gemma4uv) + audio (gemma4ua) encoders |
mtp-gemma-4-12b-it-uncensored.gguf |
444 MB | Multi-token prediction (gemma4-assistant, 4 next-N layers, Q8_0 weights) for speculative decoding |
Usage
CPU Only
llama-server -m gemma-4-12B-it-uncensored-KleidiAI.gguf \
-mm mmproj-gemma-4-12B-it-bf16.gguf \
--alias "Gemma-4-Uncensored" --jinja \
-lm mmap -c 65535 --top-k 40 --temp 1.0 --top-p 0.95 --min-p 0 --repeat-penalty 1.1 --presence-penalty 0
Dedicated GPU
llama-server -m gemma-4-12B-it-uncensored-KleidiAI.gguf \
-mm mmproj-gemma-4-12B-it-bf16.gguf \
--model-draft mtp-gemma-4-12b-it-uncensored.gguf \
--spec-type draft-mtp --spec-draft-n-max 3 \
--alias "Gemma-4-Uncensored" --jinja -fit on \
-lm mmap -c 65535 --top-k 40 --temp 1.0 --top-p 0.95 --min-p 0 --repeat-penalty 1.1 --presence-penalty 0
Minimal offload
llama-server -m gemma-4-12B-it-uncensored-KleidiAI.gguf \
-mm mmproj-gemma-4-12B-it-bf16.gguf \
--model-draft mtp-gemma-4-12b-it-uncensored.gguf \
--spec-type draft-mtp --spec-draft-n-max 3 -ngld 0 -ngl 0 \
--alias "Gemma-4-Uncensored" --jinja -fit off \
-lm mmap -c 65535 --top-k 40 --temp 1.0 --top-p 0.95 --min-p 0 --repeat-penalty 1.1 --presence-penalty 0
Notes
- Quantization is a hand-tuned Q4_0/Q8_0 mix targeting KleidiAI fast CPU kernels, not a plain quantize pass.
- Model metadata embeds sampling defaults: top_k 64, top_p 0.95, temp 1.0.
- This quantization is only suitable for ARM systems. Using Q4_K_S is better for all other architectures.
Uncensoring
This is based off of TrevorJS's excellent work (https://huggingface.co/TrevorJS/gemma-4-12B-it-uncensored),
and carries the same refusal rates:
| Before | After | |
|---|---|---|
| Refusals (mlabonne, 100 prompts) | 99/100 | 6/100 |
| Refusals (cross-dataset, 686 prompts) | — | 14/686 (2.0%) |
| KL Divergence | 0 (baseline) | 0.0556 |
| Quality (coherence) | — | no degradation (response audit + Q8 inference verified) |
Cross-Dataset Validation
Tested against 4 independent prompt datasets to verify generalization:
| Dataset | Prompts | Refusals |
|---|---|---|
| JailbreakBench | 100 | 2/100 |
| tulu-harmbench | 320 | 4/320 |
| NousResearch/RefusalDataset | 166 | 4/166 |
| mlabonne/harmful_behaviors | 100 | 4/100 |
| Total | 686 | 14/686 (2.0%) |
How this differs from vanilla heretic
- Norm-preserving biprojection instead of standard projection (preserves weight magnitudes)
- Per-layer refusal directions instead of one global direction
- Deterministic single-pass instead of 50-trial Optuna search (faster, same or better results)
- LoRA merge before save for clean GGUF-compatible tensor names
Usage disclaimer
This model is distributed as-is, under the terms of the apache-2.0 license. It's minimal refusal rates make it ideal for research and user freedom, yet the responsibility for it's usage falls upon you, that serves the model.