language:

• en
  license: mit
  size_categories:
• 100K<n<1M
  task_categories:
• text-classification
  tags:
• ai-safety
• red-teaming
• agentic-security
• tool-use
• prompt-injection
• smolagents
• guardrails
  pretty_name: Bartholomew Agentic Red-Team Evaluation Benchmark

Bartholomew Agentic Red-Team Evaluation Benchmark (btp-agent-redteam-evals)

This benchmark provides 105,000 curated, ground-truth labeled evaluation samples of autonomous agent tool calls, covering prompt injections, destructive shell breakouts, catastrophic SQL database mutations, SSRF cloud metadata attacks, and credential leaks.

Dataset Details

• Total Invariant Vectors: 105,000 test cases covering Shell, SQL, Network SSRF, Credentials, and Benign Operations.
• Synthesized & Curated Invariants: Generated from real-world agent security incidents and compiler AST fuzzing (test_million_invariant_fuzz.py).
• Data Shards:
  • data/train-00000-of-00003.jsonl (35,000 vectors)
  • data/train-00001-of-00003.jsonl (35,000 vectors)
  • data/train-00002-of-00003.jsonl (35,000 vectors)
• Fields:
  • id: Unique identifier (BTP-EVAL-000001 through BTP-EVAL-105000)
  • payload: Proposed tool invocation, bash string, or SQL query
  • threat_category: Attack classification (shell_destructive, sql_ddl_wipe, reverse_shell, ssrf_metadata, secret_leak, benign_shell, etc.)
  • expected_verdict: DENY or ALLOW
  • is_malicious: Boolean ground truth
  • rule_id: Bartholomew BTP invariant classification ID

How to Evaluate Your Agent Guard

from datasets import load_dataset
from btp_guard import Guard

# Automatically streams or loads all 105,000 sharded vectors
dataset = load_dataset("acnbartholomew/btp-agent-redteam-evals", split="train")
guard = Guard(strict=True)

correct = 0
for row in dataset:
    res = guard.check(row["payload"])
    expected = (row["expected_verdict"] == "ALLOW")
    if res["allowed"] == expected:
        correct += 1

accuracy = (correct / len(dataset)) * 100
print(f"Evaluated {len(dataset)} invariant vectors. Overall Accuracy: {accuracy:.2f}%")

Links & Resources

• GitHub Repository: ivegotahunnitonit/bartholomew
• Live Leaderboard: acnbartholomew/agent-guardrails-leaderboard
• Interactive Simulator: acnbartholomew/bartholomew-agent-guard
• PyPI Library: pip install btp-guard