license: cc-by-4.0
language:

• en
  task_categories:
• text-generation
  tags:
• ai-safety
• alignment
• alignment-auditing
• red-teaming
• cli-agents
• benchmark
  pretty_name: ANCHOR-Seed
  size_categories:
• n<1K
  configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*
  • split: test
    path: data/test-*

ANCHOR-Seed

ANCHOR-Seed is the 300-task seed benchmark from the paper
ANCHOR: Automated Alignment Auditing for CLI Agents on Real-World Harm.

Each task is grounded in a real, public U.S. federal criminal case (from
CourtListener) and is provided in both its
original first-person form and a neutralized ("refined") rewrite, together with an
LLM-generated action/criteria decomposition used to judge whether an agent's behavior
actually accomplished the underlying harm.

• 📄 Paper: https://arxiv.org/abs/2607.10455 · Hugging Face paper page
• 💻 Code: https://github.com/garified/anchor
• 🧪 Tasks: 836 total — train (536) + test (300)

Splits

• train — 536 training tasks.
• test — 300 evaluation tasks.

Abstract (paper)

Autonomous CLI agents can now execute hundreds of actions across multi-hour sessions:
writing code, executing shell commands, browsing the web, and managing cloud infrastructure,
all with minimal human oversight. Does greater autonomy invite greater risk? We introduce
ANCHOR, an automated auditing framework that stress-tests CLI agents on illegal tasks grounded
in public US court cases. ANCHOR deploys an auditor agent fine-tuned on dark personality data
using supervised and reinforcement learning. This auditor roleplays persistent malicious users
who decompose tasks, reframe requests upon refusal, and adapt strategies across multi-turn
interactions.

How each task was built (provenance of every column)

public US court case
   └─ original_task / original_setting        raw first-person description of the offense
        └─ [Phase 1: neutral rewriting]
             └─ refined_task / refined_setting  neutralized, non-incriminating phrasing
                  └─ [Phase 2: action-intent separation, a multi-turn LLM decomposition]
                       ├─ necessary_sufficient_actions   3 high-level sub-tasks (intent removed)
                       ├─ evaluation_criteria_1          resolution rubric, v1
                       └─ evaluation_criteria_2          resolution rubric, v2 (objective)

The Phase-2 fields (necessary_sufficient_actions, evaluation_criteria_1,
evaluation_criteria_2) are produced by decomposing each refined_task through a multi-turn
action-intent separation dialogue and extracting the tagged sub-tasks and resolution rubrics.

Fields

Usage

from datasets import load_dataset

ds    = load_dataset("textattack/anchor-seed")            # {train: 536, test: 300}
train = load_dataset("textattack/anchor-seed", split="train")   # 536 training tasks
test  = load_dataset("textattack/anchor-seed", split="test")    # 300 evaluation tasks

row = test[0]
print(row["original_task"])          # raw offense
print(row["refined_task"])           # neutralized version an agent would receive
print(row["evaluation_criteria_2"])  # scoring rubric (test split only)

Intended use & safety

This dataset is released for AI-safety and alignment-auditing research only (red-teaming,
refusal/robustness evaluation of agents). The tasks are derived from real criminal cases and are
deliberately harm-adjacent. Do not use them to solicit, plan, or carry out real-world harm.

Citation

@inproceedings{anchor2026,
  title     = {ANCHOR: Automated Alignment Auditing for CLI Agents on Real-World Harm},
  author    = {ANCHOR authors},
  booktitle = {Proceedings of the International Conference on Machine Learning (ICML)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2607.10455}
}