This was an experiment to better understand specifically the KDA layers and their impact on refusals. The entire model was frozen other than the KDA output projection. It's fairly clear this has an impact but only when using far too powerful of an intervention, and the effect is only mild at best. It's a usable checkpoint, but I'm leaving this up mostly just as a curiosity.
KL Divergence: 0.19 (This is rediculous, expected is around 1e-6 to 1e-5)
Refusals: 215/619