pretty_name: CrackedShell-Sec-171
license: apache-2.0
tags:

• security
• benchmark
• agents
• red-team
• skill-bundle
  size_categories:
• 100<n<1K

CrackedShell-Sec-171

CrackedShell-Sec-171 is a file-native benchmark corpus for evaluating agent security behavior, runtime safeguards, tool-use controls, and red-team robustness. The dataset ships each benchmark as a standalone bundle so evaluators can preserve the original multi-file layout during ingestion and execution.

Highlights

• 171 benchmark bundles with per-case metadata, labels, and supporting assets.
• Multi-file bundle format designed for agent-skill ingestion, runtime-policy testing, and adversarial evaluation pipelines.
• Structured annotations exposed through manifest.json, labels.json, and optional file_graph.json files.
• Broad scenario coverage across families such as benign (30), unknown (19), data_exfiltration (18), prompt_injection (17), credential_theft (16), agent_hijacking (15).
• Severity-style annotations present in the corpus with values including critical (63), high (57), none (33), medium (17), low (1).

Repository layout

.
├── README.md
├── LICENSE
└── skill_bundles/
    ├── index.json
    ├── BND-*/
    │   ├── manifest.json
    │   ├── labels.json
    │   ├── file_graph.json
    │   └── files/
    │       ├── SKILL.md
    │       ├── docs/
    │       ├── runtime/
    │       └── scripts/

Data format

• skill_bundles/index.json: corpus-level index for the full benchmark set.
• manifest.json: case metadata such as family, severity, permissions, entrypoints, and benchmark annotations.
• labels.json: per-bundle labels for evaluation and filtering.
• file_graph.json: optional graph representation of relationships between files inside a bundle.
• files/: benchmark payload content, including instructions, scripts, runtime configs, prompts, and documentation assets.

Recommended use

This corpus is well suited for:

• agent security benchmarking
• policy and guardrail evaluation
• adversarial skill-ingestion tests
• tool permission and runtime-control validation
• benchmark-driven red-team experiments

Citation

If you use CrackedShell-Sec-171 in your work, please cite:

@misc{shiroyang2026skillssecurity,
  author = {Shiro Yang and Rio Yang and Andrew Chen and Pony Ma and {Mind Lab}},
  title = {OpenClaw Skill Security: Intent-Capability-Behavior Consistency as a New Framework},
  year = {2026},
  howpublished = {Mind Lab: A Lab for Experiential Intelligence},
  note = {https://macaron.im/mindlab/research/openclaw-skill-security-intent-capability-behavior-consistency-framework}
}

License

This dataset is released under the Apache License 2.0. You may use, modify, and redistribute the corpus under the terms of the license; see the LICENSE file for the full text.

Because some benchmark bundles include executable scripts and adversarial test content, they should only be inspected or run in isolated evaluation environments with appropriate safeguards.