language:
- en
license: gemma
base_model: google/diffusiongemma-26B-A4B-it
tags: - abliteration
- arbitrary-rank-ablation
- heretic
- uncensored
- mixture-of-experts
- block-diffusion
DiffusionGemma 26B - A4B IT (Arbitrary-Rank Ablation)
This model is a decensored/abliterated version of google/diffusiongemma-26B-A4B-it, created using the Heretic framework's Arbitrary-Rank Ablation (ARA) algorithm.
Abliteration Details
DiffusionGemma-26B relies heavily on deep Mixture-of-Expert (MoE) layers (up to 128 experts) and a block-diffusion paradigm. Ablating the deepest layers causes significant catastrophic degradation and KL divergence spikes in this specific architecture. To achieve optimal safety-vector suppression while preserving the model's complex structural integrity, the ablation was strictly capped at layer 20.
This model represents the Pareto-optimal Trial 144 from a 150-trial Optuna hyperparameter search, balancing minimal refusal rates with extremely low KL divergence.
ARA Hyperparameters (Trial 144)
- Target Layers: 1 to 20
- Abliterated Components:
attn.o_projandmlp.down_proj - Preserve Good Behavior Weight: 0.8357
- Steer Bad Behavior Weight: 0.0359
- Overcorrect Relative Weight: 0.0418
- KNN Neighbor Count: 11
Evaluation Metrics
- Initial Refusals: 98 / 100 (Harmless Alpaca / Harmful Behaviors)
- Final Refusals: 4 / 100
- KL Divergence: 0.1106
Usage
This model behaves exactly like the base diffusiongemma-26B-A4B-it model but lacks synthetic refusal boundaries. It can be used via standard Transformers loading:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Umranz/diffusiongemma-26B-A4B-it-ara"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")