license: other
license_name: mixed-per-config
task_categories:

• text-classification
  language:
• en
• multilingual
  tags:
• prompt-injection
• jailbreak
• llm-security
• benchmark
• evaluation
  pretty_name: Prompt Injection Eval Suite
  configs:
• config_name: notinject
  data_files:
  • split: test
    path: data/notinject.parquet
• config_name: xstest
  data_files:
  • split: test
    path: data/xstest.parquet
• config_name: orbench_hard
  data_files:
  • split: test
    path: data/orbench_hard.parquet
• config_name: deepset_test
  data_files:
  • split: test
    path: data/deepset_test.parquet
• config_name: jackhhao_test
  data_files:
  • split: test
    path: data/jackhhao_test.parquet
• config_name: simsonsun_jailbreaks
  data_files:
  • split: test
    path: data/simsonsun_jailbreaks.parquet
• config_name: boundary_pairs_test
  data_files:
  • split: test
    path: data/boundary_pairs_test.parquet
• config_name: agentic5k_test
  data_files:
  • split: test
    path: data/agentic5k_test.parquet
• config_name: neuralchemy_test
  data_files:
  • split: test
    path: data/neuralchemy_test.parquet
• config_name: slabs_test
  data_files:
  • split: test
    path: data/slabs_test.parquet
• config_name: bipia
  data_files:
  • split: test
    path: data/bipia.parquet
• config_name: piarena
  data_files:
  • split: test
    path: data/piarena.parquet
• config_name: llmail_phase2
  data_files:
  • split: test
    path: data/llmail_phase2.parquet
• config_name: synthetic_docs_test
  data_files:
  • split: test
    path: data/synthetic_docs_test.parquet
• config_name: synthetic_direct_test
  data_files:
  • split: test
    path: data/synthetic_direct_test.parquet
• config_name: synthetic_v2_test
  data_files:
  • split: test
    path: data/synthetic_v2_test.parquet

Prompt Injection Eval Suite

This is the evaluation suite behind the
Prompt Injection Guard model card, packaged so
anyone can reproduce the comparison or evaluate their own detector. It covers over-defense (false alarms on benign
text that looks suspicious), direct injection and jailbreaks, and indirect injection planted in documents, emails and
RAG contexts.

Every row has text, label (1 = injection or jailbreak, 0 = benign), source, kind (direct / indirect) and lang.
Under this suite's definition, plain harmful requests with no attempt to override instructions are benign. That is
why XSTest's "unsafe" prompts are labelled 0.

Notes:

• Each config keeps its upstream license. Please cite and respect the original datasets. BIPIA-derived rows are
  CC-BY-SA-4.0, so derivatives must keep that license.
• bipia and piarena are derived: we built positive/negative pairs from the upstream contexts. See
  code/data/build_v0.py in the model repo.
• The synthetic_* configs were generated by us and are in-distribution for our own models. Treat them as
  supplementary.
• Two more sets used on the model card are not included, because they are gated and non-commercial:
  qualifire/prompt-injections-benchmark and Mindgard/evaded-prompt-injection-and-jailbreak-samples.

Results for 9 open detectors: Prompt Injection Detector Leaderboard.

Quick evaluation

from datasets import load_dataset
from transformers import pipeline
clf = pipeline("text-classification", model="Horizon-Labs/prompt-injection-guard-base", truncation=True)
ds = load_dataset("Horizon-Labs/prompt-injection-eval-suite", "notinject", split="test")
preds = [int(p["label"] == "injection") for p in clf(ds["text"], batch_size=32)]
print("accuracy", sum(int(p == y) for p, y in zip(preds, ds["label"])) / len(preds))