license: mit
task_categories:
- text-generation
tags: - preference-data
- dpo
- reward-modeling
- evaluation-awareness
- chain-of-thought
IFEval Obfuscation — Full Preference Pairs (2023 constitution)
Preference pairs over responses from a Wood-Labs eval-aware 49B organism (nemotron-nas / DeciLM),
judged under the 2023 Claude constitution, for training reward models / DPO on verbalized
evaluation-awareness (VEA). These are the FULL files the RMs actually trained on — not the
earlier filtered subset.
Files (DPO-ready)
prefs_2023_leak_full.jsonl— 14,074 pairs. Judge saw the CoT + answer ("leak" visibility).prefs_2023_noleak_full.jsonl— 13,967 pairs. Judge saw the answer only ("no-leak" visibility).
Each row: prompt, chosen, rejected (raw completions incl. <think>…</think>), plus per-pair labels:chosen_vea, rejected_vea, vea_contrast, vea_on_rejected, chosen_metagaming,rejected_metagaming, metagaming_contrast, chosen_if_score, rejected_if_score,p_prefer_chosen, visibility, strict_matched_contrast, length_matched.
Two different "contrast" counts — read this
There are two notions of contrast; they differ by ~13×:
| definition | leak | no-leak |
|---|---|---|
strict_matched_contrast=True (same prompt, length-matched, only the VEA clause differs) |
~292 | ~296 |
vea_contrast=True (one side's CoT verbalizes eval-awareness, the other's doesn't) — the real signal |
3,782 (27%) | 3,765 (27%) |
The ~150–290 figure is the strict flag (a tiny curated eval subset). The training signal is ~3,800
VEA-contrast pairs. Filter with vea_contrast == True.
The directional signal (what a model actually learns)
Count isn't enough — what matters is whether VEA is on the rejected side:
| VEA-rejected (anti) | VEA-chosen (pro) | % anti | net | |
|---|---|---|---|---|
| leak (judge saw CoT) | 2,310 | 1,472 | 61.1% | +838 |
| no-leak (judge saw only answer) | 2,021 | 1,744 | 53.7% | +277 |
- VEA lives 99.4% in the CoT, ~0% in the answer. Every VEA-contrast = "one CoT verbalizes
eval-awareness, the other doesn't; both answers clean." - Leak carries a real, moderate anti-VEA lean (61%) because the judge saw the CoT.
- No-leak is ~coin-flip (54%) because the CoT was invisible to the judge — so a no-leak-trained
RM/DPO learns only weak answer-side correlates of VEA, not VEA itself.
DPO usage
from datasets import load_dataset
ds = load_dataset("json", data_files="prefs_2023_leak_full.jsonl", split="train")
ds = ds.filter(lambda r: r["vea_contrast"]) # 3,782 anti/pro-VEA pairs
# columns prompt / chosen / rejected are DPO-ready as-is
For the strongest anti-VEA training, filter vea_on_rejected == True (VEA always dispreferred) or
oversample it to push the directional signal above the natural 61%.
Counts: see CONTRAST_COUNTS.json.