language:
- en
license: cc-by-4.0
size_categories: - n<1K
task_categories: - other
pretty_name: SkillHarm
tags: - ai-safety
- agent-security
- prompt-injection
- skill-based-attacks
- red-teaming
- llm-agents
configs: - config_name: fpp
data_files: fpp_index.jsonl - config_name: smp
data_files: smp_index.jsonl
SkillHarm
Lifecycle-Aware Skill-Based Attacks via Automated Construction
📄 Paper · 🌐 Project Page · 💻 GitHub · 🤗 Data
Agent skills occupy a privileged position in the agent workflow — agents are expected to implicitly follow and execute them — which makes third-party skills a vulnerable supply-chain attack surface. SkillHarm is a benchmark of skill-based attacks across the skill-use lifecycle, paired with a systematic taxonomy of 12 skill-relevant risks. Every attack is grounded in a runnable Harbor task environment and scored by a deterministic attack-success-rate (ASR) evaluator.
The benchmark contains 879 attack samples across two attack scenarios:
| Scenario | Samples | Tasks | Skills | Risks |
|---|---|---|---|---|
| Fixed-Payload Poisoning (FPP) — a fixed poisoned skill directly compromises any task session that invokes it | 687 | 57 user tasks | 71 | 12 |
| Self-Mutating Poisoning (SMP) — an initially benign execution silently mutates persistent skill content, deferring harm until a subsequent reuse | 192 | 12 task pairs | 6 | 12 |
Risk taxonomy
12 risk types (the risk_id field) across 3 categories:
| Category | Risk types |
|---|---|
| Data Pipeline Exploitation | data_exfiltration, output_manipulation, poisoning |
| System Environment Exploitation | privilege_escalation, unauthorized_file_modification, backdoor_injection, dos, malware_deployment, system_corruption |
| Agent Autonomy Exploitation | goal_hijacking, anti_forensics, proxy_attack |
Loading
This dataset is a tree of runnable task environments, not a flat table. Download the whole release with the Hub client:
from huggingface_hub import snapshot_download
local_dir = snapshot_download(repo_id="osunlp/SkillHarm", repo_type="dataset")
MANIFEST.json indexes all 879 samples; each entry's path points at the corresponding sample directory inside the snapshot.
fpp_index.jsonl and smp_index.jsonl mirror MANIFEST.json (one row per sample) and load directly for browsing or filtering:
from datasets import load_dataset
fpp = load_dataset("osunlp/SkillHarm", "fpp", split="train") # 687 rows
smp = load_dataset("osunlp/SkillHarm", "smp", split="train") # 192 rows
Running a sample
Each sample is a runnable Harbor task. Please check our github repo for the end-to-end driver.
FPP — run the user task with the poisoned skill, then read the deterministic ASR signal:
harbor run -p fixed-payload-poisoning/samples/<task>/<target>/<risk>/<designer>/<sample_id> \
-a <agent> -m <model>
The path segments are the MANIFEST fields task_id / target_slug / risk_id / design_agent_id / sample_id — or just pass an entry's path directly. ASR is computed by tests/run_asr_evaluator.py (installed into the container by tests/test.sh).
SMP — run Task A with the modified shared skill, snapshot the skill folder, overlay it into Task B's container, run Task B, then check test_detection.py.
Citation
@article{ning2026skillharm,
title = {SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction},
author = {Ning, Yuting and Zhang, Zhehao and Lal, Yash Kumar and Gou, Boyu
and Li, Junyi and Ruan, Weitong and Ye, Chentao and Gupta, Rahul
and Yang, Diyi and Su, Yu and Sun, Huan},
journal={arXiv preprint arXiv:2606.02540},
year = {2026}
}