license: apache-2.0
license_link: https://ai.google.dev/gemma/docs/gemma_4_license
pipeline_tag: text-generation
library_name: transformers
tags:
- heretic
- uncensored
- abliteration
- diffusion
- moe
base_model: google/diffusiongemma-26B-A4B-it
Note:This project represents independent research conducted on personal compute resources (rented from Modal.com) and is not associated with my employer or organization
diffusiongemma-26B-A4B-it-HERETIC-Uncensored
26.6B params · 4B active · MoE · Diffusion · Apache-2.0 ·
This is the first abliteration of DiffusionGemma 26B A4B, produced using heretic with custom patches to support its block-diffusion architecture and MoE expert layers.
DiffusionGemma is not a standard autoregressive transformer, so this required significant engineering work that hasn't been done before for this model class.
Usage
Load using the DiffusionGemmaForBlockDiffusion class directly, not AutoModelForCausalLM:
import torch
from transformers import AutoTokenizer, DiffusionGemmaForBlockDiffusion
model_id = "edwixx/diffusiongemma-26B-A4B-it-HERETIC-Uncensored"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = DiffusionGemmaForBlockDiffusion.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="cuda"
)
messages = [{"role": "user", "content": "Your prompt here"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to("cuda")
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=200, do_sample=True, temperature=0.7)
response = tokenizer.decode(out.sequences[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)
Results
| Base model | google/diffusiongemma-26B-A4B-it |
| Method | Heretic (directional ablation + LoRA + Optuna TPE) |
| Trials run | 200 |
| Best trial | #89 |
| Refusals | 13/100 (down from 100/100) |
| KL Divergence | 0.4909 |
What had to be patched
Heretic assumes standard autoregressive models. DiffusionGemma needed several custom changes:
Expert-Granular Abliteration (EGA): The MoE experts.down_proj is a batched parameter [128, 2816, 704], not a regular linear layer. Heretic skips it by default. We iterate over all 128 expert slices per layer and apply norm-preserving biprojected ablation to each one. Without this, refusals barely moved. Credit to TrevorS for the original EGA idea on Gemma 4.
Weight tying fix: The encoder and decoder share the exact same weight tensors (confirmed via data_ptr). PEFT only wraps the encoder side, so the decoder wouldn't see the LoRA delta during generation. Fixed with a context manager that temporarily merges the LoRA into the shared base weights before each generation call.
Task type: DiffusionGemma's generation mixin doesn't implement prepare_inputs_for_generation, which the default CAUSAL_LM PEFT task type requires. Switched to FEATURE_EXTRACTION.
Hidden states: DiffusionGemma's generate() doesn't support output_hidden_states. Switched to forward hooks on encoder layers to capture per-layer activations for the refusal direction PCA.
Output handling: The model returns DiffusionGemmaGenerationOutput with a .sequences attribute, not a raw tensor like standard models. Patched all heretic output handling.
Notes
This is a research release. The model will attempt to answer prompts it previously refused. 13/100 sensitive prompts still trigger refusals in testing.
Known issue: "own" tokens
Some token positions in generated outputs show the word "own" where other content was expected.
Example output:
"It pulls the own with own hands" (should be "It pulls the tide with gentle hands")
Initially assumed to be a diffusion denoising fallback, but kabachuha pointed out this artifact shows up in base autoregressive Gemma4 models too when weights are pushed hard (e.g. overfit LoRA training). Likely a Gemma4-level placeholder token, not specific to the diffusion architecture. A small LoRA fine-tune on clean data would probably reduce it. PRs welcome.
Citation
@misc{edwixx-diffusiongemma-26B-A4B-it-HERETIC-Uncensored,
author = {Anurag Kanade},
title = {diffusiongemma-26B-A4B-it-HERETIC-Uncensored},
year = {2026},
publisher = {Hugging Face},
journal = {Hugging Face Hub},
howpublished = {\url{https://huggingface.co/edwixx/diffusiongemma-26B-A4B-it-HERETIC-Uncensored}}
}