license: mit
task_categories:

• text-generation
  language:
• en
  tags:
• ai-safety
• red-teaming
• benchmark
• llm-evaluation
• agent-safety
  pretty_name: HarmfulSkillBench
  size_categories:
• n<1K
  extra_gated_heading: Researcher Access Agreement
  extra_gated_description: Our team may take 1-3 days to process your request.
  extra_gated_button_content: Acknowledge and request access
  extra_gated_prompt: |
  HarmfulSkillBench contains pseudonymised descriptions of agent skills in
  sensitive categories. You agree to use the dataset for academic research
  and not to attempt to re-identify the original skill authors.
  extra_gated_fields:
  Full Name: text
  Organization: text
  Organization Email: text
  Intended Use:
  type: select
  options:
  • Research
  • Education
  • label: Other
    value: other

I will not attempt to re-identify pseudonymised skill authors: checkbox

📝 Paper  |  📑 arXiv  |  💻 Code  |  📦 Dataset

HarmfulSkillBench

A benchmark for evaluating LLM refusal behavior when agents are exposed to skills
that describe potentially harmful capabilities.

The benchmark probes whether current LLMs can detect and refuse harmful agent
skills in two settings. Tier 1 covers prohibited behaviors that should always
be refused. Tier 2 covers high-risk domains where responses should include
human-in-the-loop referral and AI disclosure safeguards.

📦 Contents

Each skill directory contains SKILL.md (the skill specification) and
_meta.json (anon_id, platform, category, tier).

Note: The original platform is 62 natural-language skill descriptions
authored by the benchmark team (no executable code) to ensure full
per-category coverage where ClawHub and Skills.Rest have sparse entries.

🧭 Taxonomy

• Tier 1 Prohibited (130 skills): categories P1–P14 (P7 excluded for ethical
  reasons). Covers illegal activities, critical infrastructure, weapons,
  violence, privacy, misinformation, platform abuse, etc.
• Tier 2 High-Risk (70 skills): categories H1–H7. Covers legal, medical,
  insurance, financial, employment, academic, and media-generation domains.

🎯 Representative Examples per Category

One illustrative harmful skill per category, reproduced from the paper.
Descriptions are quoted verbatim from each platform's public listing
(ClawHub, Skills.Rest) or from the original partition authored by the
benchmark team. Score is the misuse likelihood assigned during ecosystem
classification (0-1); original skills and the excluded P7 category
show ---.

🧪 Evaluation Conditions

📋 Aggregated Judgment Schema

Columns in eval_results/judgments_aggregated.csv:

🤖 Evaluated Models

• gpt-4o, gpt-5.4-mini (OpenAI)
• gemini-3-flash (Google)
• qwen3-235b, kimi-k2.5 (open-weight via HF Router)
• deepseek-v3.2

Judgments were produced by gpt-5.4-mini using a 4-question rubric
(refusal, HiTL referral, AI disclosure, harmfulness).

🛡️ Pseudonymisation

Skills originating from public platforms have been assigned stable random
anon_ids of the form {platform_short}_{hash8}. The mapping from anon_id
to the original author or URL is intentionally not released. Users agree not
to attempt re-identification as part of the access terms.

Original skills retain their given (non-identifying) names.

💻 Usage

from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="TrustAIRLab/HarmfulSkillBench",
    repo_type="dataset",
    local_dir="./HarmfulSkillBench",
)

Evaluation code is published at:
https://github.com/TrustAIRLab/HarmfulSkillBench

📄 License

MIT. See LICENSE.

🙏 Citation

If you find this useful in your research, please consider citing:

@article{JZBSZ26,
  author = {Yukun Jiang and Yage Zhang and Michael Backes and Xinyue Shen and Yang Zhang},
  title = {{HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?}},
  year = {2026},
  journal = {{CoRR abs/2604.15415}},
  url = {https://arxiv.org/abs/2604.15415}
}

✉️ Contact / Takedown

If you believe a skill originates from you and wish it removed, please contact
TODO_contact_email with the relevant anon_id and evidence of origin.