license: cc-by-4.0
pretty_name: MonitoringBench
size_categories:
- 1K<n<10K
task_categories: - text-classification
tags: - ai-safety
- ai-control
- agentic-ai
- monitoring
- red-teaming
configs: - config_name: default
data_files:- split: test
path: master.parquet
- split: test
MonitoringBench
A benchmark for evaluating LLM-based monitors of agentic AI systems. Contains
2,644 successful attack trajectories in which an AI agent accomplished one of four harmful side tasks
(sudo escalation, firewall disabling, malware download, password leaking) in
a sandboxed Linux environment under the control_arena framework.
Each trajectory is scored by a panel of 13+ LLM monitors (GPT-3.5 / 4.x / 5.x,
Claude Opus 4.x and Sonnet 4.x, o3, o4-mini, gpt-5-nano), with both
full-trajectory ("ft_") and stepwise ("sw_") scoring variants. Attack
trajectories were generated by claude-opus-4-5 and claude-opus-4-6
following several attack-generation strategies (human-authored, automated
reconnaissance, opus-only, and refined variants of each).
Files
master.parquet— primary research artifact. 2,644 rows × 135 columns
including monitor scores, attack metadata, side-task outcomes, and a manual
taxonomy of structural / evasion / technical attack strategies.final_dataset.zip— rawinspect_ai.evallog archives organized by
side task (firewall/,malware/,password_leak/,sudo/).
Loading
from datasets import load_dataset
ds = load_dataset("neur26anonsub/ctrldataset2026", split="test")
Citation
Anonymized for NeurIPS 2026 submission.