license: other
task_categories:
- text-classification
tags: - prompt-injection
- jailbreak-detection
- false-positive
- security
- guardrails
pretty_name: Real-World Benign Use Cases (Prompt-Injection FP Benchmark)
Real-World Benign Use Cases
A curated set of 178 real-world, 100%-benign examples (label == 0 for every row) pulled from
production AI-coding-agent traffic — chat messages, tool output, shell commands, code snippets —
built specifically to stress-test prompt-injection / jailbreak classifiers for false positives.
Every row was independently judged benign with high confidence before inclusion. This is not a
random sample of production traffic: rows were preferentially drawn from content that had already
tripped up at least one classifier, then deduplicated with sentence embeddings (cosine similarity)
to remove near-identical examples. Treat the numbers below as relative comparisons between
models on hard cases, not as any single model's overall real-world FP rate.
Columns
| column | description |
|---|---|
text |
the raw content |
ground_truth |
ground truth; always 0 (benign) in this release |
<model>_pred |
binary prediction (1 = flagged as injection, 0 = benign) |
Models evaluated
| column | model |
|---|---|
wolf-defender_pred |
patronus-studio/wolf-defender-prompt-injection |
deberta-v3-base_pred |
protectai/deberta-v3-base-prompt-injection-v2 |
PIGuard_pred |
leolee99/PIGuard |
distilbert_pred |
fmops/distilbert-prompt-injection |
mdeberta-v3-base_pred |
proventra/mdeberta-v3-base-prompt-injection |
promptguard_pred |
codeintegrity-ai/promptguard |
sentinel-v2_pred |
rogue-security/prompt-injection-jailbreak-sentinel-v2 |
sentinel-v3_pred |
internal candidate model (not published) |
Results — false-positive rate on benign-only data
Since every row's true label is benign, any 1 prediction is a false positive.
| model | n | FP | FP rate |
|---|---|---|---|
| sentinel-v3 | 178 | 1 | 0.56% |
| sentinel-v2 | 178 | 26 | 14.61% |
| mdeberta-v3-base | 178 | 26 | 14.61% |
| promptguard | 178 | 28 | 15.73% |
| deberta-v3-base | 178 | 39 | 21.91% |
| PIGuard | 178 | 49 | 27.53% |
| wolf-defender | 178 | 63 | 35.39% |
| distilbert | 178 | 157 | 88.20% |