pretty_name: "Refusal ladder transcripts (base → instruct, with transcoder adapters)"
language: [en]
tags: [transcoder-adapters, gemma-2, refusal, transcripts, interpretability]
source_datasets: [walledai/HarmBench]
generated_by_models: [google/gemma-2-2b, google/gemma-2-2b-it, Qwen/Qwen3-8B]
Refusal ladder: transcripts
Transcripts from the refusal-ladder evaluation of 2026-08-10. The same 93 prompts (63 strict-flip harmful HarmBench requests + 30 benign control requests) were run through a ladder of models that each carry more instruction-tuning machinery, all rendered with the identical gemma-2-2b-it chat template, greedy-decoded for 256 tokens, and labelled by the Qwen/Qwen3-8B judge. The gap between base_plus_attn and the adapter arms isolates what the trained sparse transcoder contributes to refusal with attention held fixed (+57 pp on harmful prompts).
Files
Two folders, one per run (the "experiment-name"):
| folder / experiment-name | Slurm job | contents |
|---|---|---|
refusal_ladder/ |
16717433 | harm_<id>.md (63) + benign_<id>.md (30); each file holds the request and, per arm, the judge label/confidence/rationale and the full response for all six arms below |
refusal_ladder_hybrid_ftonly/ |
16717626 | same 93 prompts, only the corrected hybrid_ft arm (GemmaScope replacement restricted to the fine-tuned layers 0/24/25); its labels were grafted into the main results.json |
| arm | model | description |
|---|---|---|
base |
google/gemma-2-2b |
un-tuned base model |
base_plus_attn |
adapter checkpoint with its transcoder decoder zeroed | instruct attention/embeddings/LayerNorm + base MLPs, no transcoder |
hybrid_ft |
base model with GemmaScope transcoders (instruct-fine-tuned at layers 0/24/25, siddharthmb/2026.TA.gemma2_2b_gemmascope_transcoders_instruct_ft_L0-24-25) replacing those MLPs |
can the fine-tuned base transcoders alone carry the forward pass |
adapter_tc8192 |
siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793860 |
deployed sparse transcoder adapter |
adapter_huge |
siddharthmb/2026.TA.gemma2_2b_huge_tc16384_decb_l1w0.0003_norm_sch_tarbb_lb2.0_ln1.0_dr500000_lr2e-04_bs8_sl |
16384-feature adapter |
instruct |
google/gemma-2-2b-it |
instruction-tuned target |
Regenerate
GPU node (jagupard), repo root, LARGE_ARTIFACTS_DIR + HF_TOKEN exported:
uv run --no-sync python experiments/refusal_tracing/scripts/eval_refusal_ladder.py --max_new_tokens 256 --batch_size 8 --judge_batch_size 8 --output_dir experiments/refusal_tracing/results/refusal_ladder
uv run --no-sync python experiments/refusal_tracing/scripts/eval_refusal_ladder.py --arms hybrid_ft --hybrid_replace_layers finetuned --max_new_tokens 256 --output_dir experiments/refusal_tracing/results/refusal_ladder_hybrid_ftonly
uv run --no-sync python experiments/refusal_tracing/scripts/eval_refusal_ladder.py --report_only --output_dir experiments/refusal_tracing/results/refusal_ladder # re-render tables + chart, no GPU
Cluster paths
- Slurm logs:
logs/interesting_queries/20260810_142433_16717433.{out,err},logs/interesting_queries/20260810_144334_16717626.{out,err}(checkout/juice2/u/siddharth/transcoder-adapters). - In git:
experiments/refusal_tracing/results/refusal_ladder{,_hybrid_ftonly}/{results.json,summary.md,per_prompt_labels.md,refusal_ladder.png}.
Restore into the repo
uv run hf download siddharthmb/2026.TA.experiments.refusal_tracing.refusal_ladder --repo-type dataset --include "refusal_ladder/*" --local-dir /tmp/rl && mv /tmp/rl/refusal_ladder experiments/refusal_tracing/results/refusal_ladder/transcripts
uv run hf download siddharthmb/2026.TA.experiments.refusal_tracing.refusal_ladder --repo-type dataset --include "refusal_ladder_hybrid_ftonly/*" --local-dir /tmp/rl && mv /tmp/rl/refusal_ladder_hybrid_ftonly experiments/refusal_tracing/results/refusal_ladder_hybrid_ftonly/transcripts
Provenance
- Code:
experiments/refusal_tracing/scripts/in the transcoder-adapters repo; writeup: experiments/refusal_tracing/writeup.md. - Aggregates (results.json with every record incl. judge label, confidence and rationale; summary.md tables; figures) stay in git next to the
transcripts.txtbreadcrumb that points here. The transcript.mdfiles are a human-readable rendering ofresults.jsonrecords. - Judge:
Qwen/Qwen3-8B, greedy, labels REFUSAL / COMPLIANCE / INCONCLUSIVE / GIBBERISH with a confidence and a one-sentence rationale. - Prompts:
walledai/HarmBench(standardconfig) harmful requests, idsharm_NNN= row index; the "strict flip" subset (63 ids) comes from the strict_compliance_refusal screen. - No Weights & Biases runs (generation + judging only).