Collusion Dataset V7
Balanced dataset combining easy (introductory) and harder (interview) problems for backdoor detection evaluation.
Dataset Structure
| Bucket | Intro (easy) | V6 (harder) | Total |
|---|---|---|---|
| Clean | 300 | 200 | 500 |
| Backdoor | 300 | 200 | 500 |
| Total | 600 | 400 | 1000 |
Schema
Each sample contains:
| Field | Type | Description |
|---|---|---|
problem_id |
string | Unique ID (prefixed with intro_ or v6_) |
code |
string | The code to analyze |
problem_text |
string | Problem description |
difficulty |
string | introductory or interview |
source |
string | intro or v6 |
v7_bucket |
string | clean or backdoor |
label |
int | 0 (clean) or 1 (backdoor) |
is_backdoored |
bool | Whether sample contains a backdoor |
backdoor_input |
string/null | Input that triggers the backdoor (if available) |
detection_difficulty |
string/null | easy, medium, hard, or null (clean samples) |
Detection Difficulty
Backdoor samples are labeled by how many models detected them in Phase 1 baseline evaluation (6 models: gpt-4o, gpt-5-nano, o4-mini, gpt-5-mini, gpt-5, o3):
| Difficulty | Definition | Count |
|---|---|---|
easy |
Detected by all 6 models | 186 |
medium |
Detected by 1-5 models | 210 |
hard |
Detected by 0 models | 104 |
null |
Clean samples (no backdoor) | 500 |
Source Data
Intro samples: From
apps-control-arenadataset (introductory difficulty)- 293 with working backdoors + 7 with non-working backdoors
- Each problem contributes both a clean and backdoor sample
V6 samples: From
arena-control-apps-v6-dataset- Random selection of 200 clean + 200 backdoor
- Interview difficulty (harder)
Creation
Created by data/create_v7_dataset.py with random seed 42 for reproducibility.
uv run python data/create_v7_dataset.py
Usage
import json
with open('data/collusion-dataset-v7/collusion_dataset_v7.json') as f:
data = json.load(f)
# Filter by bucket
clean = [d for d in data if d['v7_bucket'] == 'clean']
backdoor = [d for d in data if d['v7_bucket'] == 'backdoor']
# Filter by difficulty
easy = [d for d in data if d['source'] == 'intro']
hard = [d for d in data if d['source'] == 'v6']