Coffee & AI

How this model was abliterated

This model was abliterated with a custom setup built in
https://github.com/CoffeeVampir3/Mojo-Gemma4-CPU/tree/ablating, running
entirely on CPU with no low-rank projections anywhere. Everything you need to
reproduce it is in the repo. You can run the result with this repo on CPU or
just drop it into any normal inference engine — it's a regular abliterated
checkpoint, nothing special about loading it.

The overall approach is close to heretic.
Here's how it works, and where it diverges from heretic's current style.

What came out

• KL divergence on benign prompts: about 0.00026 (~0.026%) — basically the
  model's normal behavior barely moved.
• Refusals on the harmful test set: 104 down to 7.

save_abliterated: wrote checkpoints/gemma-4-26B-A4B-it-abliterated
  lambda* 2.09375: KL(full) 0.0002577923426088091
  refusals@64 7/104 (baseline 104)
  wrote checkpoints/gemma-4-26B-A4B-it-abliterated/abliteration_results.txt

Finding the direction and the per-layer schedule

First we run the model over a batch of harmful prompts and a batch of harmless
ones, and record the residual activations at every layer. Before averaging
anything, we winsorize each activation — clamp the outliers down to the 99.5th
percentile — so a handful of crazy dimensions don't end up steering the whole
thing.

Then for each layer we take the difference of means (harmful minus harmless) and
normalize it. That difference vector is the refusal direction for that layer —
and yeah, it's a separate direction per layer, not one shared direction.

We also work out a signal-to-noise ratio for each layer: how big that difference
is relative to the activation norm, scaled so the strongest layer sits at the
top. That gives us a "layer schedule" — basically how much editing each layer
should get, relative to the layer that matters most.

The edit itself

Next we snapshot the baseline: the unedited model's first-token output on a set
of harmless prompts. That's our reference point for measuring KL later.

The edit picks one global strength number and hands each layer its own slice of
it — a layer's actual strength is that global number times its schedule weight,
capped so no single layer goes overboard. At that strength, we strip the layer's
refusal direction out of the three weights that write into the residual stream:
o_proj, down_proj, and experts_down. The removal is norm-preserving, so
we're redirecting rather than shrinking — the overall magnitudes stay put.

Everything here runs on the full weights.

To feel out the right strength, we crank it up step by step and watch the KL
divergence against the baseline. Each try applies the edit, measures KL, then
puts the original weights back — so the trials don't pile up on each other, and
every one starts from a clean model.

Up to this point it's basically the same as heretic. The big difference: heretic
derives its direction low-rank, while everything here happens on the full
weights, so there's no low-rank error sneaking in.

Dialing in the strength

We treat finding the strength as a root-finding problem over a semi-fixed grid —
a grid search first, then a bisection to tighten it up.

• Grid scan: sweep the strength across a fixed grid (1.0 to 3.0) and bracket
  the crossover — the biggest value still under budget, and the smallest one
  over it.
• Bisection: squeeze that bracket down toward the crossover.
• Take the biggest strength we've confirmed under budget. That's the winner.

The nice part is every check only needs the first token's output, not a whole
generation, so the search stays cheap. KL basically hands us all the signal we
need: edit harder and the benign-prompt drift climbs right alongside it,
monotonically, so that one number is enough to steer the whole thing.

And that's it

Once we've got the crossover strength, we apply that edit to the weights and
save it out. There's your abliterated model.