license: apache-2.0
language:
- en
- ja
base_model: - google/gemma-4-E4B-it
tags: - gemma4
- Abliterated
- GGUF
- vision
- audio
gemma4-E4B-it-Abliterated-GGUF
GGUF quantizations of an abliterated build of google/gemma-4-E4B-it
(8.61B params / 42 layers / text + vision + audio).
What this is
- Method: heretic v2.2.0, ARA (Arbitrary-Rank Ablation),
Optuna TPE 100 trials on a single NVIDIA L4 (24 GB). - Trial selection: Heretic's default selection (lexicographic sort that puts refusal rate first)
picks trial 30 — refusal 0/100 but KL divergence 10.69, i.e. the output distribution has collapsed.
Trial 39 was re-selected from the Pareto front instead and is the model published here. - No fine-tuning, no merge: the abliterated checkpoint is converted to GGUF as-is.
Benchmark
| Metric | base | this model |
|---|---|---|
Refusal (mlabonne/harmful_behaviors, 100 prompts) |
100/100 | 7/100 |
KL divergence (mlabonne/harmless_alpaca, vs. base) |
— | 0.0899 |
PPL (wikitext-2-raw, llama.cpp, Q8_0) |
52.3834 ±0.64 | 50.2741 ±0.61 (0.960×) |
PPL was measured with llama-perplexity -c 2048 --chunks -1 -ngl 99 --seed 42 over 144 chunks
(294,912 tokens), base and abliterated under identical conditions. The ratio of 0.960 means the
abliterated model is slightly lower than the base — i.e. no measurable capability loss on this metric.
Files
| File | Quant | Size | SHA256 |
|---|---|---|---|
gemma4-E4B-it-Abliterated-BF16.gguf |
BF16 | 15.05 GB | a0acd28ad54eb83fa1f3dbe4fa04ad038ddc2b1aa7b684c3a3eec56234b0c660 |
gemma4-E4B-it-Abliterated-Q8_0.gguf |
Q8_0 | 8.03 GB | 99272d84fdaad6c2a32928282a3cfe4eb77524d62249b30c9e4229c28768b9fa |
gemma4-E4B-it-Abliterated-Q4_K_M.gguf |
Q4_K_M | 5.34 GB | d6fbf8cb829241646077d0376d4710ce90f539f93ca9b31f790bfce74ed97446 |
mmproj-gemma4-E4B-it-Abliterated-BF16.gguf |
BF16 (vision/audio projector) | 0.99 GB | 8fe029392ca5c550d626cdc43bd1be35073aeac861a26da46d3c07c46697796e |
The mmproj file is only needed for image / audio input. Text-only use needs just one of the model files.
Notes on the quants:
- All GGUFs are converted from the same BF16 abliterated checkpoint; the Q8_0 is bit-identical to a
plainllama-quantize ... Q8_0run on the BF16, except forper_layer_model_proj.weight, which
current llama.cpp keeps at BF16. - In Q4_K_M, llama.cpp keeps the per-layer embedding tensors (
token_embd,per_layer_token_embd)
at Q6_K, so the file is 5.34 GB at 5.66 BPW rather than the ~4.5 GB a naive 4-bit estimate suggests.
Usage (llama.cpp)
Requires a llama.cpp build with Gemma 4 support (gemma4 architecture). --jinja is required —
the chat template embedded in the GGUF is a custom one.
# Server
llama-server -m gemma4-E4B-it-Abliterated-Q8_0.gguf --jinja --temp 1
# CLI (thinking ON / OFF)
llama-cli -m gemma4-E4B-it-Abliterated-Q8_0.gguf --jinja -cnv -rea on --temp 1
llama-cli -m gemma4-E4B-it-Abliterated-Q8_0.gguf --jinja -cnv -rea off --temp 1
# Vision / audio (mmproj required)
llama-mtmd-cli -m gemma4-E4B-it-Abliterated-Q8_0.gguf \
--mmproj mmproj-gemma4-E4B-it-Abliterated-BF16.gguf --jinja -p "Describe this image."
Sampling follows the official generation_config.json (temperature 1.0 / top_k 64 / top_p 0.95),
as is standard for the Gemma family.
For the OpenAI-compatible API, set chat_template_kwargs.enable_thinking to false to disable thinking.
Recommended VRAM
| VRAM | Recommended |
|---|---|
| 16 GB+ | BF16 |
| 10 GB | Q8_0 |
| 7 GB | Q4_K_M |
Actual usage depends on context size and GPU offloading.
License
This model is a modified version of Gemma 4 E4B, released under the Apache License 2.0.