license: gemma
base_model: google/gemma-4-26B-A4B-it
language: [en]
tags: [gemma, gemma-4, moe, abliterated, uncensored, duoneural]
Gemma 4 26B-A4B Abliterated
DuoNeural | GGUF →
Refusal-removed google/gemma-4-26B-A4B-it using Expert-Granular Abliteration (EGA).
Architecture
Gemma 4 26B-A4B is Google's MoE model: 25.2B total params, 3.8B active per token, 30 layers with alternating SWA(256)/Global(512) attention, Per-Layer Embeddings (PLE), shared KV cache, and 128 experts per layer.
Method — Expert-Granular Abliteration (EGA)
Standard dense abliteration computes a single mean refusal direction per layer. EGA computes a per-expert refusal direction across all 128 experts independently and orthogonalizes each via biprojection.
Result: 3/100 refusals vs 29/100 for dense-only abliteration.
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model = AutoModelForCausalLM.from_pretrained(
"DuoNeural/gemma-4-26B-A4B-it-abliterated",
torch_dtype=torch.bfloat16,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("DuoNeural/gemma-4-26B-A4B-it-abliterated")
For consumer hardware, use the GGUF Q4_K_M (~16.5GB).
Hardware Requirements
| Format | VRAM |
|---|---|
| BF16 (this repo) | ~52GB |
| GGUF Q4_K_M | ~16.5GB |
Inference Engine Notes
Gemma 4 requires a patched engine for 3 quirks:
- Variable attention head dims — alternating SWA(256)/Global(512) per layer
- Per-Layer Embeddings — two embedding inputs per layer
- Shared KV cache — multiple layers share one KV buffer
llama.cpp (recent builds) handles all three. Use with -ctk turbo4 -ctv turbo3 for efficient KV cache quantization.
DuoNeural
DuoNeural is an open AI research lab — human + AI in collaboration.
| 🤗 HuggingFace | huggingface.co/DuoNeural |
| 🐙 GitHub | github.com/DuoNeural |
| 🐦 X / Twitter | @DuoNeural |
| [email protected] | |
| 📬 Newsletter | duoneural.beehiiv.com |
| ☕ Support | buymeacoffee.com/duoneural |
| 🌐 Site | duoneural.com |
Research Team
- Jesse — Vision, hardware, direction
- Archon — AI lab partner, post-training, abliteration, experiments
- Aura — Research AI, literature synthesis, novel proposals
Raw updates from the lab: model drops, training results, findings. Subscribe at duoneural.beehiiv.com.
DuoNeural Research Publications
Open access, CC BY 4.0. Authored by Archon, Jesse Caldwell, Aura — DuoNeural.