license: cc-by-4.0
task_categories:

• text-classification
  language:
• tr
• en
  tags:
• multilingual-guardrail-benchmark
• turkish-jailbreak-dataset
• over-refusal-dataset
• prompt-injection
• jailbreak
• guardrail
• llm-security
• red-teaming
• turkish
  pretty_name: PolyGuardBench
  size_categories:
• n<1K
  configs:
• config_name: all
  default: true
  data_files:
  • split: train
    path: data/all/train.jsonl
• config_name: axis-prompt-injection
  data_files:
  • split: train
    path: data/axis-prompt-injection/train.jsonl
• config_name: axis-jailbreak
  data_files:
  • split: train
    path: data/axis-jailbreak/train.jsonl
• config_name: axis-over-refusal
  data_files:
  • split: train
    path: data/axis-over-refusal/train.jsonl
• config_name: lang-tr
  data_files:
  • split: train
    path: data/lang-tr/train.jsonl
• config_name: lang-en
  data_files:
  • split: train
    path: data/lang-en/train.jsonl

PolyGuardBench — a multilingual guardrail benchmark (Turkish + English)

PolyGuardBench is one canonical multilingual guardrail benchmark that consolidates
several previously scattered guardrail datasets into a single, unified, de-duplicated schema.
It covers three evaluation axes — prompt injection, jailbreak, and over-refusal —
across Turkish and English, so a guardrail model or a moderation API can be scored on
attack detection and false-positive (over-refusal) behavior from one place.

It bundles a Turkish jailbreak dataset and an over-refusal dataset with English
prompt-injection and jailbreak prompts, all reduced to the same fields.

Honesty / scope. This is engineering evidence for guardrail evaluation, not a
certification and not legal/compliance advice. Every row is loaded from a real source
dataset (listed under Provenance); nothing here is fabricated or machine-generated.
The corpus is not independently "validated" — it is a consolidation of the author's
own curated sets. Despite the "multilingual" framing of some upstream shards, the
realized languages are currently only Turkish and English; other languages are a gap,
not a claim.

Schema

Each row (JSON Lines):

axis labels the evaluation dimension a row exercises: injection payloads →
prompt-injection; persona/DAN-style attacks → jailbreak; benign controls and their
harmful twins used to measure false refusals → over-refusal.

Counts (per axis / label / language)

Generated from data/ by scripts/build_dataset.py — do not edit by hand.

Loading

from datasets import load_dataset

# the whole benchmark
ds = load_dataset("fevziegeyurtsevenler/PolyGuardBench", "all", split="train")

# a single axis or language sub-config
tr = load_dataset("fevziegeyurtsevenler/PolyGuardBench", "lang-tr", split="train")
orf = load_dataset("fevziegeyurtsevenler/PolyGuardBench", "axis-over-refusal", split="train")

# quick guardrail-style split
attacks = [r for r in ds if r["label"] == "attack"]
benign  = [r for r in ds if r["label"] == "benign"]

Or read the JSONL directly without the datasets library:

import json
rows = [json.loads(l) for l in open("data/all/train.jsonl", encoding="utf-8")]

Provenance — what was consolidated

PolyGuardBench supersedes and absorbs the author's scattered guardrail shards. Rows
come from these canonical Hugging Face sources:

Superseded shards (already fully absorbed by the canonical sources above — kept on the
Hub only for history; verified at build time, see stats.json → supersession):

• fevziegeyurtsevenler/multilingual-prompt-injection
• fevziegeyurtsevenler/turkish-prompt-injection
• fevziegeyurtsevenler/multilingual-jailbreak
• fevziegeyurtsevenler/prompt-injection-classification

Related, intentionally NOT merged:
fevziegeyurtsevenler/guard-blindspots-tr
is a robustness set of mechanical transforms (casefold / leet / zero-width) of injection
payloads. Its originals are already in guardrail-arena; the transforms are synthetic
variants, so folding them in would inflate the unique-attack count. Use it separately for
robustness testing.

Deduplication

The build normalizes text (collapsed whitespace, case-insensitive) and drops duplicate or
empty rows. The canonical sources do not overlap each other (over-refusal prompts share no
text with the arena), so de-duplication mainly guards against internal repeats. Exact
figures are printed by the build and enforced by the tests.

Related work (external)

This benchmark is complementary to, and indebted to, prior over-refusal / safety
benchmarks. Credit to related public efforts that inspired the over-refusal axis:

• XSTest — Röttger et al., XSTest: A Test Suite for Identifying Exaggerated Safety
  Behaviours in LLMs (2023).
• OR-Bench — Cui et al., OR-Bench: An Over-Refusal Benchmark for Large Language
  Models (2024).
• LinguaSafe and other multilingual safety suites for the cross-lingual framing.

PolyGuardBench does not reuse their data; it provides an independent Turkish-centric
consolidation. Their coverage of English over-refusal is broader.

Sibling projects

Part of an open Turkish LLM-security toolkit by the same author:
prompt-canon (PyPI),
casefold-fuzz, prompt-lint, redteam-coverage-matrix,
the guardrail-arena dataset, and the unicode-threat-reveal / guardarena-live
Spaces — see github.com/fevziegeyurtsevenler.

License

• Data: CC-BY-4.0. Attribute Fevzi Ege Yurtsevenler.
• Upstream source datasets retain their own licenses; this consolidation is compatible
  with CC-BY-4.0 redistribution of the author's own material.

Citation

@dataset{yurtsevenler_polyguardbench_2026,
  author  = {Yurtsevenler, Fevzi Ege},
  title   = {PolyGuardBench: A Multilingual (Turkish + English) Guardrail Benchmark
             for Prompt Injection, Jailbreak, and Over-Refusal},
  year     = {2026},
  publisher = {Hugging Face},
  doi      = {10.5281/zenodo.XXXXXXX},
  note     = {DOI placeholder — to be minted on release},
  url      = {https://huggingface.co/datasets/fevziegeyurtsevenler/PolyGuardBench}
}

İlgili AltaySec kaynakları

• 🌐 AltaySec · Açık Kaynak Lab
• 💻 GitHub: github.com/fevziegeyurtsevenler
• 📚 Kardeş veri setleri: AltaySec/turkish-llm-injection · PolyOCRInject