license: other
task_categories:

• text-generation
  tags:
• ai-safety
• sycophancy
• activation-steering
• representation-engineering

Sycophancy experiment data

Data for the sycophancy steering experiments, split by what actually varies. Everything outside
the model folders is shared by every model; each model folder holds the replies that model
produced, which is what its probe trains on.

eval/data.json             evaluation prompts: validation + six held-out test units   (shared)
splits/*.jsonl             question-level splits, one question per split, no overlap         (shared)
probe_pool/                the TriviaQA pool and its generated wrong answers          (shared)
prompts/                   extract.jsonl and gate.jsonl, prompts with no responses    (shared)
judge_spotcheck/           blind human labels validating the belief judge             (shared)
<Model>/d_bad.jsonl        512 of the model's own sycophantic replies -> LoRA probe   (per model)
<Model>/caa_pairs.jsonl    same-question (sycophantic, correcting) reply pairs -> CAA (per model)
<Model>/stage0/            the raw pool those two are drawn from, plus judge verdicts (per model)

Models: Qwen-2.5-3B-Instruct, Qwen-2.5-7B-Instruct, Llama-3.1-8B-Instruct, Gemma-2-9B-it.
103 MB total, of which ~92 MB is stage0/.

Why the probe data cannot be shared

The over-refusal experiments pair benign prompts with a fixed, canned refusal, so one dataset
serves every model. Sycophancy does not: the probe is trained on on-policy replies — the
model's own agreements with a user's wrong answer, drawn from Stage 0 and labelled by a judge. A
direction extracted from one model's replies is not defined for another, so d_bad.jsonl and
caa_pairs.jsonl differ per model by construction.

What is shared is every prompt: the questions, their splits, the evaluation set and the two
probe prompt files. prompts/extract.jsonl and prompts/gate.jsonl are byte-identical across the
four models — the build checks this and refuses to continue otherwise — so they are stored once.

Evaluation splits (shared)

eval/data.json holds 7 units. validation selects the layer and alpha; the rest are held out.

splits/ is the question-level layer behind validation and test_belief, plus the probe splits
(train, extract, gate). Splits are disjoint by question.

Files and integrity

manifest.json lists every file with its sha256 and row count, the per-model counts and selection
rules copied from each probe manifest, and the role of each shared group. build.py regenerates
the folder from the experiment runs.

Each d_bad row carries the prompt, the model's reply, the judge verdict, the sampling index and
the token count; caa_pairs rows carry the question with both replies.

Provenance and caveats

• Questions come from TriviaQA (probe pool) and the meg-tong/sycophancy-eval trivia subset
  (evaluation); the wrong answers in probe_pool/questions.jsonl were generated by Qwen3-32B and
  filtered, and the Stage 0 verdicts come from the same model used as a judge.
• Four of the seven evaluation units — test_pushback, test_feedback, test_mc,
  capability — were copied from an earlier letter-answer pilot, and test_feedback_free was
  assembled by hand from test_feedback. No current stage regenerates them; the rest of the data
  is reproducible from probe_pool/ and splits/.
• Gemma-2-9B has Stage 0 and probe data but no steering run. It is here for completeness.
• Upstream datasets keep their own licences; this folder redistributes them for reproducibility.