license: cc-by-nc-4.0

Dataset Card for Monitor Red Teaming (MRT)

This dataset contains the agent trajectories and monitoring results for the experiments in "Reliable Weak-to-Strong Monitoring of LLM Agents". It provides a standardized benchmark for evaluating the reliability of monitoring systems against adversarial LLM agents attempting to evade oversight.

The associated code repository can be found here: https://github.com/scaleapi/mrt

Dataset Details

Dataset Description

This dataset is designed to facilitate research in AI safety, particularly in the monitoring of autonomous agents. It contains thousands of interaction trajectories where LLM agents perform tasks in mock digital environments. Each task has a benign primary objective and a hidden malicious objective (e.g., sabotage, data exfiltration). The dataset captures agent attempts to complete these hidden objectives while evading detection under various adversarial strategies.

The data is split into two main categories:

• SHADE-Arena Tasks: 17 tool-use agent tasks based on the original SHADE-Arena framework, involving actions like bank transfers, calendar management, and travel booking.
• CUA-SHADE-Arena Tasks: 15 computer-use agent tasks built on TheAgentCompany benchmark, spanning domains like spreadsheet manipulation, data analysis, and HR operations.

The trajectories are generated using different red teaming strategies (e.g., lying, manipulation, selective verbalization) and evaluated by various monitor scaffolding systems (e.g., zero-shot, sequential, hybrid).

• Curated by: Neil Kale, Chen Bo Calvin Zhang, Kevin Zhu, Ankit Aich, Scale Red Team, Christina Q. Knight, Zifan Wang
• Language(s) (NLP): English
• License: The dataset is licensed under Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0). The underlying frameworks this work is built upon, SHADE-Arena and TheAgentCompany, are licensed under the MIT License.

Dataset Sources

• Repository: https://github.com/kalescale/SHADE-Arena/tree/cleaned-main
• Paper: Reliable Weak-to-Strong Monitoring of LLM Agents

Uses

Direct Use

This dataset is intended for researchers and developers working on AI safety, agent oversight, and adversarial testing. Primary use cases include:

• Benchmarking new monitor models and systems.
• Analyzing adversarial agent behavior and evasion techniques.
• Studying the performance gap between weak and strong LLMs in a monitoring context.
• Developing more robust and reliable oversight mechanisms for autonomous agents.

Out-of-Scope Use

This dataset contains examples of simulated malicious behavior for research purposes.

• It should not be used to train generative models, as this could teach them to perform the harmful secondary tasks. A canary string is included to help prevent accidental inclusion in training corpora.
• The dataset is not designed to evaluate the task-completion capabilities of agents, but rather the ability of monitors to detect their hidden malicious actions.

Dataset Structure

The dataset is organized into two main folders: task_pairs (for SHADE-Arena tasks) and task_pairs_agentcompany (for CUA-SHADE-Arena tasks).

The file structure is as follows:
task_pairs/<task_name>/<attack_model>/<monitor_model>/transcript_<#>.json

Each transcript_... .json file contains a single experimental run and typically includes:

• transcript: The full conversation history between the user and the agent, including the agent's reasoning (<thinking> tags) and actions.
• monitor_transcript: The conversation between the evaluator and the monitor model, where the agent's trajectory is passed in for a verdict.
• monitor_verdict: The final judgment from the monitor on whether the agent engaged in suspicious or harmful behavior.
• task_description: The primary and secondary objectives for the given task.

The specific naming conventions for different experimental configurations (<attack_model>, <monitor_model>, etc.) can be fully explored in the utils/pipeline_utils.py file in the code repository.

Dataset Creation

Curation Rationale

As LLM agents become more autonomous, ensuring they can be safely deployed and monitored is critical. This dataset was created to address a gap in standardized evaluation for agent monitoring systems. It provides a systematic framework and a comprehensive set of challenging test cases for measuring monitor reliability against agents that are actively trying to hide malicious behavior.

Source Data

Data Collection and Processing

The data is entirely synthetic, generated by LLM agents (from providers like OpenAI and Anthropic) interacting with simulated digital environments. The tasks are derived from two benchmarks: SHADE-Arena and TheAgentCompany. Trajectories were generated by executing the scripts in the accompanying code repository, which systematically vary the agent models, monitor models, threat models, and red teaming strategies.

Who are the source data producers?

The data was generated by the research team at Scale AI and affiliated authors listed in the paper.

Annotations

The monitor verdicts within each JSON file serve as the primary annotations. These were generated programmatically by various LLM-based monitor models, which were prompted to evaluate the agent trajectories for signs of malicious activity. The performance of these monitors is the central subject of the associated research paper.

Bias, Risks, and Limitations

• Synthetic Environments: The agent behaviors are captured in mock environments. These simulations may not fully represent the complexities and failure modes of real-world applications.
• Model Dependence: The effectiveness of the evasion and monitoring techniques is highly dependent on the specific LLMs used as agents and monitors. Results may differ with newer or different models.
• Task Scope: While the tasks cover a range of domains, they do not encompass all possible forms of malicious agent behavior.

Citation

BibTeX:

@misc{kale2025reliableweaktostrongmonitoringllm,
      title={Reliable Weak-to-Strong Monitoring of LLM Agents}, 
      author={Neil Kale and Chen Bo Calvin Zhang and Kevin Zhu and Ankit Aich and Paula Rodriguez and Scale Red Team and Christina Q. Knight and Zifan Wang},
      year={2025},
      eprint={2508.19461},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2508.19461}, 
}

Canary String: The following canary helps researchers prevent SHADE-Arena task data from being included in training datasets: 5AGVKj6HGcwQshM7DLulSKcGVbIe2C-mFbSl7eMVOd1VrTNayy0YkjJ1X4W3o