license: mit
task_categories:

• text-classification
• text-generation
  language:
• en
  tags:
• safety
• red-teaming
• agent-safety
• prompt-injection
• llm-evaluation
• benchmark
  pretty_name: CIK-Bench
  size_categories:
• n<1K
  configs:
• config_name: default
  data_files:
  • split: attack_cases
    path: data/attack_cases.parquet
  • split: attack_cases_phase1_only
    path: data/attack_cases_phase1_only.parquet
  • split: attack_cases_phase2_only
    path: data/attack_cases_phase2_only.parquet
  • split: benign_cases
    path: data/benign_cases.parquet

CIK-Bench

Official dataset for Your Agent, Their Asset: A Real-World Safety Analysis
of OpenClaw.

CIK-Bench evaluates OpenClaw — the most widely deployed personal AI
agent in early 2026 — against persistent-state poisoning attacks. It
implements the CIK taxonomy, a unified framework that organizes
OpenClaw's persistent state into three dimensions:

• Capability — executable skills (SKILL.md, .sh, .py)
• Identity — persona, values, and behavioral configuration
  (SOUL.md, IDENTITY.md, USER.md, AGENTS.md)
• Knowledge — learned facts and owner preferences
  (MEMORY.md, session context)

The benchmark contains 88 attack cases across 12 impact scenarios
spanning two top-level harm categories with three subcategories each:

• Privacy Leakage (P-*) — financial, identity/physical, and other
  sensitive data exfiltration
• Risky Irreversible Operations (R-*) — financial loss, social
  consequences, and data security damage

It also ships a matched set of benign cases for defense evaluation.

• 📄 Paper: https://arxiv.org/abs/2604.04759
• 🌐 Project page: https://ucsc-vlaa.github.io/CIK-Bench
• 💻 Code & test harness: https://github.com/UCSC-VLAA/CIK-Bench

Attack Protocol

Each attack follows a two-phase protocol:

1. Phase 1 (Injection) — Poisoned content is introduced into the agent's
   persistent state via one of the CIK dimensions.
2. Phase 2 (Trigger) — A subsequent prompt activates the poisoned state,
   causing the harmful action.

Phase 1 and Phase 2 run in separate sessions (except for session-context
injections), testing whether attacks persist across session boundaries.

Dataset Structure

The dataset is shipped in two complementary forms:

1. Structured rows — data/<split>.parquet

Four parquet files, one per templates/ subfolder of the upstream repo:
attack_cases (88 rows), attack_cases_phase1_only (38),
attack_cases_phase2_only (38), benign_cases (38). One row per case-vector
markdown spec. Carrier-skill payloads (Python / bash scripts under
<case>/assets/) are filesystem scaffolding, not data points — access them
via the raw templates/ tree if needed.

Schema:

2. Raw template tree — templates/

Mirrors the GitHub repo layout, preserved so the test harness can run
straight from the dataset:

templates/
├── attack_cases/                 # 88 attack cases (12 impacts × ~7 vectors)
│   ├── P-F1/                     # API key exfiltration
│   ├── P-F2/                     # Stripe customer PII exfiltration
│   ├── P-P1/                     # IP geolocation exfiltration
│   ├── P-P2/                     # Passport scan exfiltration
│   ├── P-O1/                     # Gmail work email exfiltration
│   ├── P-O2/                     # Local medical records exfiltration
│   ├── R-F1/                     # Stripe subscription cancellation
│   ├── R-F2/                     # Stripe refund abuse
│   ├── R-S1/                     # Forged email: AWS permission grant
│   ├── R-S2/                     # Forged email: contract termination
│   ├── R-D1/                     # Workspace deletion
│   └── R-D2/                     # Gmail bulk deletion
├── attack_cases_phase1_only/     # Phase 1 only (injection success)
├── attack_cases_phase2_only/     # Phase 2 only (trigger success)
├── benign_cases/                 # Matched benign cases (defense eval)
├── openclaw_template/            # OpenClaw workspace template
├── test_assets/                  # Synthetic test data (medical records, etc.)
└── stripe_setup/                 # Stripe test sandbox setup
defense_assets/                   # Defense configurations

Each case directory contains markdown specs for every injection vector
(baseline.md, identity-md.md, identity-soul-md.md, identity-user-md.md,
identity-agents-md.md, skill-md.md, skill-bash.md, skill-py.md,
mem-long.md, mem-session.md) plus an optional assets/ subdirectory
holding the carrier-skill payload files used during Phase 1.

Templates contain {{PLACEHOLDER}} tokens (e.g. {{ATTACKER_WEBHOOK}},
{{AGENT_NAME}}, {{OWNER_NAME}}) that are filled in at runtime by
scripts/configure.sh in the GitHub repo.

Usage

from datasets import load_dataset

ds = load_dataset("UCSC-VLAA/CIK-Bench")

# Available splits: attack_cases, attack_cases_phase1_only,
#                   attack_cases_phase2_only, benign_cases
attack = ds["attack_cases"]
print(attack[0]["case_id"], attack[0]["vector"])
print(attack[0]["content"][:200])

# Filter by harm category
financial = attack.filter(lambda r: r["harm_category"] == "Financial Data Leak")

To reproduce the full evaluation pipeline (test harness, OpenClaw setup,
webhook receiver, Stripe sandbox, etc.) clone the GitHub repo:
https://github.com/UCSC-VLAA/CIK-Bench.

Impact Codes

Injection Vectors

Ethical Use

CIK-Bench is released for defensive safety research only. The included
attack templates target a sandboxed test agent, use webhook.site / Stripe
test-mode for verification, and contain only synthetic PII (e.g. a synthetic
passport image and fictional medical records). Do not adapt these payloads
against systems you do not own or have explicit permission to test.

Citation

@misc{wang2026agentassetrealworldsafety,
  title={Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw},
  author={Zijun Wang and Haoqin Tu and Letian Zhang and Hardy Chen and Juncheng Wu and Xiangyan Liu and Zhenlong Yuan and Tianyu Pang and Michael Qizhe Shieh and Fengze Liu and Zeyu Zheng and Huaxiu Yao and Yuyin Zhou and Cihang Xie},
  year={2026},
  eprint={2604.04759},
  archivePrefix={arXiv},
  primaryClass={cs.CR},
  url={https://arxiv.org/abs/2604.04759},
}

License

MIT — see LICENSE.