license: apache-2.0
language: [tr, en]
task_categories: [text-classification]
tags: [prompt-injection, llm-security, ai-security, guardrails, over-refusal, red-teaming, turkish, benchmark, jailbreak]
pretty_name: "guardrail-arena (EN+TR guardrail benchmark)"
size_categories: [n<1K]
configs: [{config_name: default, data_files: data.jsonl}]
guardrail-arena — a two-axis, multilingual guardrail benchmark
from datasets import load_dataset
ds = load_dataset("fevziegeyurtsevenler/guardrail-arena")
Score any LLM prompt-injection guardrail on two axes at once — does it catch attacks
(miss-rate down) and does it avoid over-blocking benign traffic (over-refusal down) — in
English and Turkish. A guardrail is any callable text -> 0|1.
337 prompts: 217 injections (EN 110 + TR 107) · 80 plain benign · 40 security-adjacent
benign — legitimate text that discusses or quotes attacks (the over-refusal trap most
benchmarks miss).
Seed leaderboard (reproducible)
| Guardrail | Miss-rate down | Over-refusal (all) down | Over-refusal: security-adjacent down | F1 up | TR miss |
|---|---|---|---|---|---|
| protectai-deberta-v2 | 0.02 | 0.18 | 0.40 | 0.94 | 0.02 |
| AltaySec-detector* | 0.00 | 0.23 | 0.70 | 0.94 | 0.00 |
| jackhhao-jailbreak | 0.46 | 0.12 | 0.25 | 0.67 | 0.83 |
| keyword | 0.88 | 0.12 | 0.38 | 0.20 | 0.91 |
| regex-rules | 0.91 | 0.10 | 0.30 | 0.16 | 0.90 |
Finding 1 — over-refusal cliff: nobody over-blocks plain requests, but on security-adjacent
text over-refusal jumps to 25–70%. Even ProtectAI's widely-deployed detector over-blocks 4 of 10
legitimate security prompts. Finding 2 — multilingual blind spot: an English-trained jailbreak
classifier misses 83% of Turkish attacks. Over-defense and monolingual bias are first-class
failure modes.
* AltaySec-detector has train-set overlap on injections + plain benign; every other guard
(including the ProtectAI and jackhhao open models) is fully held-out.
Schema
text (str) · label (0 benign / 1 injection) · language (en/tr) · category · source.
Run it yourself
Harness, baselines, and the "add your guardrail" workflow:
👉 github.com/fevziegeyurtsevenler/guardrail-arena
Related AltaySec resources
- 🕵️ uncloak — invisible prompt-injection scanner
- 🤗 turkish-prompt-injection-detector — the model baseline
- 📚 prompt-injection-detection-rules · multilingual-prompt-injection
- 🌐 AltaySec · Açık Kaynak Lab
Citation
@misc{yurtsevenler2026guardrailarena,
title={guardrail-arena: A Two-Axis, Multilingual Benchmark for LLM Guardrails},
author={Yurtsevenler, Fevzi Ege}, year={2026}, publisher={AltaySec},
howpublished={\url{https://github.com/fevziegeyurtsevenler/guardrail-arena}}
}
Apache-2.0 · Türkçe-first AI/LLM security by AltaySec.