pretty_name: CompoSkill-Bench
tags:

• llm-agents
• agent-security
• prompt-injection
• skill-composition
• red-teaming
• benchmark
• ai-safety
  language:
• en
• zh
  size_categories:
• 1K<n<10K
  task_categories:
• text-generation

CompoSkill-Bench

A benchmark of compositional skill chain attacks against LLM agents.

CompoSkill-Bench is the companion dataset of the paper
CompoSkill: Compositional Skill Chain Attacks from Individually Scanner-Passing LLM Agent Skills.
It contains 1,140 long-horizon professional workflow records built from real marketplace skill ecosystems,
covering 5 threat categories and 6 professional scenarios, evaluated on two agent platforms
(OpenClaw and Nanobot).

The benchmark targets a systemic gap in agent skill certification: a skill may pass every per-skill
security scanner individually, yet participate in a risky composition when an agent connects its
outputs, capabilities, or side effects with those of other scanner-passing skills. Skill-composition
risk is therefore a path-level property rather than a node-level property.

Dataset Summary

• Records: 1,140
• Threat categories: 5
• Professional scenarios: 6
• Agent platforms: OpenClaw, Nanobot
• Attack settings: white-box (victim skill pool known) and black-box (role profile only)
• Headline results (from the paper): risk Chain Formation Rate (CFR) up to 83.3% (white-box)
  and 80.6% (black-box), while existing skill scanners intercept only a limited fraction of the
  risky compositions; a bridge-bonus-then-hop-decay pattern is observed (attack success decreases
  once risk chains exceed three skills).

Threat Categories

Professional Scenarios

Each threat is instantiated across six long-horizon professional workflows:

Repository Structure

Each record is a complete agent workspace snapshot:

<threat>/
├── common/<scenario>/            # base workspace configuration
│   ├── task.md                   # long-horizon professional task specification
│   ├── skills_manifest.yaml      # manifest of the installed skill pool
│   ├── runtime.yaml              # agent runtime configuration
│   ├── AGENTS.md / SOUL.md / ... # agent persona & workspace files
│   ├── skills/                   # installed marketplace skills (individually scanner-passing)
│   ├── data/clean_data/          # environment files: configs, logs, documents
│   └── <sub_agent_workspace>/    # per-role agent workspaces with their own skills & data
└── variants/<scenario>/          # attack-variant configurations

Environment files (e.g., CI/CD pipelines, cluster configs, audit logs, backups manifests) are
synthetic and self-contained, so agents can execute the workflows end-to-end in a sandbox.

Intended Uses

• Benchmarking skill scanners against composition-level risk (node-level certification is shown to be insufficient).
• Red-teaming research on LLM agent skill marketplaces and tool ecosystems.
• Developing and evaluating defenses against skill chain attacks and indirect prompt injection via skills.

This dataset is intended for defensive security research and education.

Ethical Considerations

• All workspaces, skills, logs, and documents are synthetic or sanitized; the dataset contains no
  real user credentials, personal data, or live service endpoints.
• Malicious skills included in the records are research artifacts for benchmarking purposes.
  Do not deploy them against real systems or production agents.
• The associated paper follows responsible disclosure practices; vulnerabilities discovered in real
  products during the research were reported to the affected vendors.

Citation

If you use CompoSkill-Bench, please cite:

@misc{liu2026composkill,
      title={CompoSkill: Compositional Skill Chain Attacks from Individually Scanner-Passing LLM Agent Skills},
      author={Mingxiao Liu and Zhoumian Jiang and Jianan Ma and Jian Zhang and Jialuo Chen and Xinhao Deng and Zhen Wang},
      year={2026},
      eprint={2608.16246},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2608.16246}
}

Contact

• Hugging Face: Limax11
• Issues and questions: please open a discussion on the dataset page.

Downloadable benchmark archives

The complete benchmark is published as five ZIP archives under data/, one for each threat category. Each archive preserves the category directory and all original files. Local .cache directories are excluded.

manifest.json lists every included file, its size and SHA-256 checksum, and the checksum of each ZIP archive.

To download and extract the benchmark:

from pathlib import Path
import zipfile
from huggingface_hub import snapshot_download

download = snapshot_download(
    repo_id="Limax11/CompoSkill-Bench",
    repo_type="dataset",
    allow_patterns=["data/*.zip", "manifest.json"],
)
destination = Path("CompoSkill-Bench")
destination.mkdir(exist_ok=True)
for archive in sorted((Path(download) / "data").glob("*.zip")):
    with zipfile.ZipFile(archive) as zipped:
        zipped.extractall(destination)