license: mit
language:

• en
  pretty_name: ActBench
  size_categories:
• 10K<n<100K
  task_categories:
• text-generation
  tags:
• benchmark
• agents
• agent-safety
• behavioral-safety
• prompt-injection
• red-teaming
• trajectories
  paperswithcode_id: actbench
  configs:
• config_name: task_pairs
  data_files:
  • split: test
    path: data/task_pairs/*.parquet
• config_name: trajectories
  data_files:
  • split: test
    path: data/trajectories/*.parquet

ActBench

ActBench is a benchmark for evaluating behavioral safety in tool-using cowork agents from execution trajectories. This dataset accompanies ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents and the ZJUICSR/ActBench source repository.

Dataset contents

This release contains two configurations:

• task_pairs: 300 rows. Each row pairs one benign task with its adversarial variant and embeds the complete public UTF-8 task bundles: YAML definitions, manifests, workspace files, fixtures, skills, logs, and graders. These 300 pairs correspond to 600 benchmark cases.
• trajectories: 24,000 rows from 20 agent-harness/model combinations. Each row contains one released trajectory, its official score summary, provenance metadata, and a SHA-256 link to the canonical full release.

The task content is intentionally preserved verbatim. It includes synthetic credentials, adversarial instructions, canary markers, test logs, and synthetic paths that are part of the benchmark semantics. The trajectory projection is derived from the canonical public release bundle prepared by the ActBench authors.

Usage

from datasets import load_dataset

task_pairs = load_dataset("ZJUICSR/ActBench", "task_pairs", split="test")
trajectories = load_dataset("ZJUICSR/ActBench", "trajectories", split="test")

print(len(task_pairs))     # 300
print(len(trajectories))   # 24000

For lower-memory iteration over trajectories:

trajectories = load_dataset(
    "ZJUICSR/ActBench",
    "trajectories",
    split="test",
    streaming=True,
)

task_pairs schema

The summary columns identify the task, scenario, behavior taxonomy, attack method, prompt, expected behavior, and attack goal. clean_files and attack_files are lists of records with path, sha256, size_bytes, and exact UTF-8 content. The original scenario.yaml, task.yaml, and manifests are also exposed directly for convenient inspection.

To reconstruct an embedded bundle:

from pathlib import Path

row = task_pairs[0]
root = Path("reconstructed") / row["task_id"] / "attack"
for file in row["attack_files"]:
    target = root / file["path"]
    target.parent.mkdir(parents=True, exist_ok=True)
    target.write_text(file["content"], encoding="utf-8")

trajectories schema

Each row includes combo_id, backend, model, suite, task_id, role (clean or attack), attack_slot, run_number, trajectory identifiers and hashes, trajectory_json, score_json, and full-release linkage fields. JSON payloads are stored as strings so their original nested schemas remain stable across different harnesses.

Safety and intended use

This dataset is intended for safety research, agent evaluation, reproducibility, and analysis of tool-using systems. It contains adversarial instructions and synthetic sensitive-looking values by design. Do not treat embedded instructions, credentials, URLs, identities, or canary values as operational data, and do not execute trajectory actions against real services without an isolated environment.

Model outputs may contain mistakes, unsafe behavior, or provider-specific artifacts. Scores should be interpreted under the ActBench evaluation protocol and the exact harness/model metadata recorded in each row.

License

The benchmark code and released dataset are provided under the MIT License. See the source repository for the license text and implementation details.

Citation

@misc{yao2026actbench,
  title={ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents},
  author={Hongwei Yao and Yiming Liu and Meihui Chen and Jieling Chen and Zikun Chen and Yiling He and Wangze Ni and Cong Wang and Kui Ren},
  year={2026},
  eprint={2608.09476},
  archivePrefix={arXiv},
  primaryClass={cs.CR},
  url={https://arxiv.org/abs/2608.09476}
}