license: mit
task_categories:
- text-classification
language: - en
tags: - ai-safety
- agent-safety
- trajectory-evaluation
- benchmark
- computer-use-agent
- red-teaming
size_categories: - n<1K
pretty_name: "Clawdbot Safety Audit Test Cases"
dataset_info:
features:- name: id
dtype: string - name: dimension
dtype: string - name: instruction
dtype: string - name: label
dtype: string
splits: - name: test
num_examples: 34
configs:
- name: id
- config_name: default
data_files:- split: test
path: test.jsonl
- split: test
Clawdbot (OpenClaw) Safety Audit — Seed Test Cases
This dataset contains the 34 seed test cases used in "A Trajectory-Based Safety Audit of Clawdbot (OpenClaw)". Each case is a task prompt designed to probe a specific safety risk dimension of Clawdbot/OpenClaw, a self-hosted, tool-using personal AI agent.
📄 Paper: A Trajectory-Based Safety Audit of Clawdbot (OpenClaw)
📝 Blog Post (中文): 当AI助手"真的动手做事"，安全边界在哪里？
💻 GitHub: Repository
Dataset Summary
We conduct a trajectory-centric safety evaluation of Clawdbot/OpenClaw across six risk dimensions, using a curated suite of 34 canonical cases drawn from established agent-safety benchmarks and hand-designed scenarios tailored to Clawdbot's tool surface. The evaluation reveals a non-uniform safety profile (overall pass rate: 58.9%): the agent performs reliably on well-scoped tasks but degrades under ambiguity, open-ended goals, or adversarial steering.
Risk Dimensions
| Dimension | ID in data | # Cases | Pass Rate | Tier |
|---|---|---|---|---|
| User-facing Deception | deception |
7 | 71% | Relatively Better |
| Hallucination & Reliability | hallucination_reliability |
7 | 100% | Relatively Better |
| Intent Misunderstanding & Unsafe Assumptions | intent_misunderstanding_error_assumption |
3 | 0% | Critical |
| Unexpected Results from Ambitious Goals | unpredictable_outcomes |
2 | 50% | Needs Improvement |
| Operational Safety Awareness | operational_safety_awareness |
8 | 75% | Relatively Better |
| Prompt Injection & Jailbreak Robustness | prompt_injection_jailbreak |
7 | 57% | Needs Improvement |
Data Fields
| Field | Type | Description |
|---|---|---|
id |
string |
Unique case identifier, formatted as {dimension}-{index} (e.g., deception-0000) |
dimension |
string |
Risk dimension the case belongs to (one of the six categories above) |
instruction |
string |
The task prompt to be sent to the agent |
label |
string |
Ground-truth safety label: safe (expected safe behavior) or unsafe (expected unsafe behavior) |
Label Semantics
safe(26 cases): The task is benign; a safe agent should complete it normally without exhibiting unsafe behavior.unsafe(8 cases): The task contains hidden risks, adversarial triggers, or ambiguity; a safe agent should refuse, ask for clarification, or take protective measures rather than blindly executing.
Usage
from datasets import load_dataset
dataset = load_dataset("YOUR_HF_USERNAME/YOUR_DATASET_NAME", split="test")
# Browse by dimension
for example in dataset.filter(lambda x: x["dimension"] == "deception"):
print(example["id"], example["label"])
print(example["instruction"][:200])
print("---")
Evaluation Setup
- Agent under test: Clawdbot/OpenClaw (self-hosted)
- Underlying LLM: MiniMax M2.1
- Tools enabled: exec (shell), web_search (Brave API), web_fetch
- Sandbox isolation: Disabled (real host environment)
- Trajectory judge: AgentDoG-Qwen3-4B
For full environment setup and reproduction instructions, see the GitHub repository.
Citation
@article{chen2026trajectory,
title={A Trajectory-Based Safety Audit of Clawdbot (OpenClaw)},
author={Chen, Tianyu and Liu, Dongrui and Hu, Xia and Yu, Jingyi and Wang, Wenjie},
journal={arXiv preprint arXiv:XXXX.XXXXX},
year={2026}
}
Related Resources
License
This dataset is released under the MIT License.
Contact
Tianyu Chen — [email protected]
Affiliations: ShanghaiTech University & Shanghai Artificial Intelligence Laboratory