license: llama3.2
base_model: meta-llama/Llama-3.2-1B-Instruct
base_model_revision: 9213176726f574b556790deb65791e0c5aa438b6
library_name: transformers
pipeline_tag: text-generation
tags:

• abliteration
• refusal-direction
• research-artifact
• llama

Llama-3.2-1B-Instruct-abliterated

A refusal-ablated edit of meta-llama/Llama-3.2-1B-Instruct (pinned
revision 9213176726f574b556790deb65791e0c5aa438b6): the readout-space
refusal direction was orthogonalized out of the untied lm_head weight,
making the edit persistent — no hook required at inference time.
First non-Qwen patient of this engine; the founding direction method
was originally characterized on Llama-2-class models, so this run
measures it back on a modern small Llama.

What it does

Refusal behavior on a fixed 64-prompt harmful set (greedy, 200 new
tokens, marker-based scorer — identical instrument to this engine's
other patients; every arm below is per-row artifact-backed):

Direction transfer across architecture families

The refusal direction's layer fingerprint moves with the architecture:
the coherence scan selects a mid-stack site (layer 9 of 16,
coherence 0.717) here, versus the deep sites of the Qwen2.5 family
(L17/24). Same method, materially different geometry — the run's main
architecture-dependence datapoint.

Benign behavior

Benign preservation on the 64-prompt harmless set, same instrument
(per-row artifacts; benign floor gate = baseline − 10pp clears at every arm):

Capability guardrail (MMLU)

0-shot MMLU over all 61 subjects (lm-eval 0.4.13, fp16, seed 0):

Loss: −0.29pp against the ≤3.0pp limit — PASS, 10× margin. The base
arm's measurement is digit-identical (48.2694) across two independent
GPU sessions; per-arm raw results + the engine log ship under eval/mmlu/.

Try it

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "sbussiso/Llama-3.2-1B-Instruct-abliterated"
tok = AutoTokenizer.from_pretrained(repo,
    clean_up_tokenization_spaces=False)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto")

messages = [{"role": "user", "content": "Explain what a for loop is."}]
prompt = tok.apply_chat_template(messages, add_generation_prompt=True,
    tokenize=False)
enc = tok(prompt, return_tensors="pt")
out = model.generate(**enc, max_new_tokens=200, do_sample=False)
print(tok.decode(out[0][enc["input_ids"].shape[1]:],
    skip_special_tokens=True))

Method

Founding directional-editing recipe (Arditi et al. 2024): 64-pair
contrastive direction extraction → coherence-ranked site selection →
weight-space orthogonalization. This variant (wd_B) edits the
lm_head readout: W ← W − (W r̂) r̂ᵀ with the untied lm_head
(tie_word_embeddings true → false, persisted in config). The edit's
disk-verified residual (max |W r̂| = 7.1e-05) agrees to three
significant digits across two GPU edit streams and a deterministic
CPU rebuild of the same edit.

Honest limitations

• Probe tables are from fixed 64-prompt sets, greedy decoding,
  single-seed, single marker-set scorer — not a benchmark-suite claim.
• Selective-refusal calibration (RefusalBench-style graded refusal) and
  refusal-residual breadth (e.g. SORRY-Bench classes) are not yet run
  on this patient.
• Any post-hoc refusal ablation is recoverable by small benign
  fine-tuning (literature finding; applies to this edit).
• Meta's Llama 3.2 Community License governs the base model and this
  edit; use under that license's terms.

References

• Arditi, A. et al. 2024. Refusal in Language Models Is Mediated by a
  Single Direction. NeurIPS 2024. https://arxiv.org/abs/2406.11717
• Xie, T. et al. 2025. SORRY-Bench: Systematically Evaluating Large
  Language Model Safety Refusal. ICLR 2025. (residual-breadth instrument
  queued for this patient)
• Muhamed, A. et al. 2025. RefusalBench: Generative Evaluation of
  Selective Refusal in Grounded Language Models. EACL 2026.
  (selective-refusal instrument queued for this patient)
• Malla, S. et al. 2025. The Geometry of Refusal: Why Post-Hoc Safety
  Is Fragile and Pretraining-Time Safety Persists.
  https://arxiv.org/abs/2609.06934

Provenance

• Base weights: meta-llama/Llama-3.2-1B-Instruct @
  9213176726f574b556790deb65791e0c5aa438b6 (gated; access accepted under
  the owner account).
• This variant (wd_B): tie_word_embeddings true → false (persisted);
  main weights sha256 cb9ad4d09bb787ac0bbe77966afa06a1e82e23535b0f02beec7d7870618bcc21
  (GPU-native — the engine's own edit stream on L4; the canonical bytes
  the run's measurements were made against). A deterministic CPU rebuild
  of the same edit was produced and verified from the shipped direction
  banks: sha256 6417be31231b76300c7013acccfd30251cc5c105356089a456950e23666e0a9c,
  behaviorally identical (probe rows digit-exact, edit residuals agree
  to 3 significant figures; bytes differ by platform — the run's
  documented device-provenance finding). Rebuild recipe + residual
  chain ship in eval/rebuild_evidence.json; the twin bytes are not
  duplicated here (regenerate from the banks instead).
• Direction banks: refusal_direction_A.npy (residual-space, ‖d‖ 3.78)
  and refusal_direction_B.npy (readout-space, ‖d‖ 74.58) — the run's
  banked stage-A artifacts, shipped for full re-derivability.
• Charts in charts/ are generated programmatically from the recorded
  probe/coherence/MMLU artifacts (make_card_charts.py ships with the run),
  in the engine's dark house palette; no hand-typed digits.
• Full method + evaluation write-ups: available on request.

Files

This is a research artifact. It is not a product and is not
intended for production use.