base_model: google/gemma-4-12B-it
library_name: transformers
license: gemma
license_name: gemma-terms-of-use
license_link: https://ai.google.dev/gemma/terms
pipeline_tag: image-text-to-text
language:

• en
  tags:
• abliterated
• uncensored
• gemma4
• refusal-removal
• weight-editing
• research

gemma4-12b-it-abliterated

A weight-edited derivative of google/gemma-4-12B-it
with the refusal direction orthogonalised out of the residual writers.

This model has no refusal behaviour. Read Intended use and
Security before loading it. If you need a model that declines
requests, use the base model instead.

Model details

Method

1. Capture the last-token residual for 150 harmful (AdvBench-style) and 150
   harmless (Alpaca) prompts at every layer.
2. d_l = normalize(mean(harmful) - mean(harmless)).
3. Orthogonalise the residual writers with a Gaussian profile centred on layers
   24 and 28, sigma 6, alpha 1.0, embedding included.
4. W' = W - alpha * g(l) * d (d^T W), with
   g(l) = exp(-(l-24)^2 / 72) + exp(-(l-28)^2 / 72); for the embedding
   E' = E - alpha (E d) d^T.
5. Save bf16. No training, no fine-tuning.

The embedding edit is not optional: without it the direction is re-injected at
every token and the rest of the edit barely registers.

Results

Scored by deepseek-flash on a three-way rubric (comply / soft-redirect /
refuse) plus actionability 0-3. Identical prompts for both models.

Large benchmark, 769 harmful / 275 benign, built from JailbreakBench, XSTest
and HarmfulQA, de-duplicated against the capture set:

Actionability on harmful: 0.75 → 1.82 of 3. On benign: 2.00 → 1.84.

AdvBench holdout (92 prompts, the set used for the ablation work):

Degenerate-output rate is identical to the base model on the same 92 harmless
prompts (9/92), so the edit adds no coherence damage.

The base model complies with 2 of 92 AdvBench prompts but 246 of 769 on the large
benchmark, so the large set is the harder test and the one to read.

Why alpha is small and the band is wide

A narrow band needs a very high alpha to remove the direction, and high alpha
mangles the model. The wide band is what reaches both low refusals and intact
coherence.

Generation

Gemma 4 ends a turn with <turn|> (token 106). It must be a stop token, or the
model starts a second turn and visibly repeats itself.

from transformers import AutoModelForImageTextToText, AutoTokenizer
import torch

mid = "pasangimhana/gemma4-12b-it-abliterated"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForImageTextToText.from_pretrained(
    mid, torch_dtype=torch.bfloat16, device_map="auto")

msgs = [{"role": "user", "content": "Explain how a bill becomes law."}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
ids = tok(text, return_tensors="pt").to(model.device)

stop = [tok.eos_token_id, tok.convert_tokens_to_ids("<turn|>")]
with torch.inference_mode():
    out = model.generate(**ids, max_new_tokens=512, do_sample=True,
                         temperature=0.7, eos_token_id=stop,
                         pad_token_id=tok.pad_token_id)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))

The chat template opens an empty thought channel before generation. That is
normal for this checkpoint and does not need to be disabled.

Intended use

Research on representation-level refusal: measuring how refusal is distributed
across layers, how weight edits propagate through normalisation, and how
evaluation instruments change the reported result.

Appropriate uses: interpretability work, red-team tooling that needs a model
without refusals, synthetic-data generation where the base model's refusals are
the obstacle.

Inappropriate uses: anything the Gemma Prohibited Use Policy forbids, any
deployment where the absence of refusal is a liability, any setting where users
have not been told the model is unfiltered.

Security

This model will comply with requests the base model refuses, including requests
that are illegal or harmful. That is the point of the edit and it cannot be
switched off at runtime.

• Do not expose it as a public endpoint without an external policy layer.
• Do not put it behind an agent with tool access and no human approval step.
• Output may contain instructions that are dangerous to follow. Treat generations
  as untrusted input, the same as any model output, but with a lower prior on
  self-restraint.
• Prompt injection is more consequential here: a model that does not refuse is a
  model that will follow an injected instruction it would otherwise decline.

No safety filtering is included, and none is planned.

Limitations

• The direction comes from 150 + 150 prompts from two corpora. A different
  distribution will move it. The benchmark is disjoint at the string level but
  not distributionally independent.
• 6 of 769 held-out harmful prompts still refuse. The ones inspected are requests
  a text model cannot literally satisfy, or ones where it answers while arguing
  the opposite premise.
• Actionability on benign prompts falls from 2.00 to 1.84. Cause not diagnosed.
• No capability benchmark (MMLU, GSM8K) was run. The coherence guard and the
  benign suites are the only evidence that general ability is intact.
• Judge calibration: 24 hand-labelled rows, Cohen's kappa 0.56 overall and 0.71
  on the harmful slice. Substantial agreement where the headline is decided, poor
  agreement on benign prompts that cannot be answered literally.

Files

License and terms

Distributed under the Gemma Terms of Use,
the same terms as the base model, including the
Prohibited Use Policy.

This is an unaffiliated derivative. Not produced, endorsed or supported by
Google. Removing refusal behaviour from the weights does not remove the
obligations attached to the weights. If you redistribute this model, or a
quantised or fine-tuned version of it, keep the notice and the terms with it.

Disclaimer

Provided as-is, without warranty of any kind, for research use. No fitness for
any purpose is claimed. The author is not responsible for outputs, downstream
fine-tunes, quantisations or deployments. If you need refusal behaviour, use the
base model.

Citation

@misc{gimhana2026gemma4abl,
  title  = {Removing the refusal direction from Gemma 4 12B},
  author = {Pasan Gimhana},
  year   = {2026},
  note   = {Weight-space abliteration, calibrated three-way judge}
}