pretty_name: "Strict base-compliance vs instruct-refusal screen: transcripts"
language: [en]
tags: [transcoder-adapters, gemma-2, refusal, transcripts, interpretability]
source_datasets: [walledai/HarmBench]
generated_by_models: [google/gemma-2-2b, google/gemma-2-2b-it, Qwen/Qwen3-8B]

Strict base-compliance vs instruct-refusal screen (HarmBench, 200 prompts): transcripts

200 transcripts (one per HarmBench standard prompt, harm_000 … harm_199) from the screen of 2026-07-30 that selected the "strict flip" prompt set used by every later refusal-tracing experiment (refusal ladder, prefill transplant, first-ten-token attribution graphs, feature ablation). For each request the instruct model google/gemma-2-2b-it generated one chat-templated continuation and the base model google/gemma-2-2b generated three continuations, one per prompt style (direct: the raw request; qa: a Question:/Answer: frame; chat: the instruct chat template), sampled with temperature 0.7 / top-p 0.9, 256 tokens max. Every continuation was labelled by the Qwen/Qwen3-8B judge. A prompt is a strict flip (63 of 200) when the instruct response is a high-confidence REFUSAL and at least one base response is a high-confidence coherent, relevant, substantive COMPLIANCE (minimum confidence 0.80).

Files

harm_<id>.md: category, whether the id was selected as a strict flip, the request, the instruct continuation with its judge outcome/confidence/rationale, then the three base continuations (direct, qa, chat) each with their judge outcome.

Regenerate

uv run --no-sync python experiments/refusal_tracing/scripts/find_strict_compliance_refusal.py --n_scan 200 --base_prompt_styles direct qa chat --max_new_tokens 256 --judge_max_new_tokens 96 --batch_size 8 --judge_batch_size 4 --base_do_sample --base_temperature 0.7 --base_top_p 0.9 --seed 0 --min_judge_confidence 0.8 --output_dir experiments/refusal_tracing/results/strict_compliance_refusal

(GPU node; the script's first log line records this invocation.)

Cluster paths

• Slurm log: logs/interesting_queries/strict_full_16396210.{out,err} (checkout /juice2/u/siddharth/transcoder-adapters).
• In git: experiments/refusal_tracing/results/strict_compliance_refusal/{results.json,summary.md}, the attribution-ready prompt files selected_prompts/ (63), selected_prompts_trimmed/ (18), comply_vs_refuse_chat/ (10), template_flip_prompts/ (3).

Restore into the repo

uv run hf download siddharthmb/2026.TA.experiments.refusal_tracing.strict_compliance_refusal --repo-type dataset --include "harm_*.md" --local-dir experiments/refusal_tracing/results/strict_compliance_refusal/transcripts

Provenance

• Code: experiments/refusal_tracing/scripts/ in the transcoder-adapters repo; writeup: experiments/refusal_tracing/writeup.md.
• Aggregates (results.json with every record incl. judge label, confidence and rationale; summary.md tables; figures) stay in git next to the transcripts.txt breadcrumb that points here. The transcript .md files are a human-readable rendering of results.json records.
• Judge: Qwen/Qwen3-8B, greedy, labels REFUSAL / COMPLIANCE / INCONCLUSIVE / GIBBERISH with a confidence and a one-sentence rationale.
• Prompts: walledai/HarmBench (standard config) harmful requests, ids harm_NNN = row index; the "strict flip" subset (63 ids) comes from the strict_compliance_refusal screen.
• No Weights & Biases runs (generation + judging only).