license: gemma
base_model: google/gemma-4-31b-it
tags:
- abliterated
- gemma4
- gemma-4
- gguf
- uncensored
- heretic
Gemma 4 31B-IT Abliterated (Q4_K_M GGUF)
Abliterated variant of google/gemma-4-31b-it with refusal behavior removed using heretic. Removes ~64% of refusals while preserving model capabilities (KL divergence 0.27).
Files
| File | Size | Description |
|---|---|---|
gemma-4-31b-it-abliterated-t126-Q4_K_M.gguf |
18 GB | Q4_K_M quantization |
Abliteration Method
- Tool: heretic v1.2.0
- Approach: Bayesian-optimized refusal direction removal via LoRA-based weight modification
- Optimization: 200 trials (60 random exploration + 140 TPE-guided), selected from Pareto front balancing refusal count vs KL divergence
- Targets: 120 modules across 60 layers (
self_attn.o_proj+mlp.down_proj) - Datasets: 400 harmful vs 400 harmless prompts (mlabonne/harmful_behaviors, mlabonne/harmless_alpaca)
- Hardware: NVIDIA H100 80GB, ~1 hour optimization
Gemma 4 Compatibility Note
Gemma 4's vision encoder uses Gemma4ClippableLinear layers not supported by PEFT. Resolved by restricting LoRA targeting to language model layers via full module paths instead of leaf name matching.
Usage
llama-server -m gemma-4-31b-it-abliterated-t126-Q4_K_M.gguf -ngl 99 --ctx-size 8192 --flash-attn on
Quantization
- Format: GGUF Q4_K_M (4.87 BPW)
- Size: ~18GB
- Target hardware: RTX 4090 (24GB), RTX 5090 (32GB), any GPU with 20GB+ VRAM