license: apache-2.0
language: [tr, en]
task_categories: [text-classification]
tags: [over-refusal, llm-safety, turkish, evaluation, xstest, guardrails, ai-security, red-teaming]
pretty_name: "Turkish Over-Refusal Set (XSTest-style)"
size_categories: [n<1K]
configs: [{config_name: default, data_files: turkish_over_refusal.jsonl}]
turkish-over-refusal-set
from datasets import load_dataset
ds = load_dataset("fevziegeyurtsevenler/turkish-over-refusal-set")
An XSTest-style over-refusal evaluation for Turkish (+English): 120 matched pairs of a benign-but-scary prompt and a refuse-worthy twin sharing the same trigger word (popcorn patlat vs nose patlat; chord vur vs shoot vur; process kill/öldür vs person). 480 prompts, 10 categories.
Finding: guards over-block Turkish, not English
| Guard | Over-refusal (benign) TR | Over-refusal (benign) EN | TR/EN |
|---|---|---|---|
| protectai/deberta-v3-base-prompt-injection-v2 | 59.2% | 0.8% | ~71x |
| fmops/distilbert-prompt-injection | 25.0% | 5.0% | 5x |
| jackhhao/jailbreak-classifier | 0.0% | 0.0% | — |
ProtectAI flags 3 of 5 benign Turkish prompts as attacks — and flags benign Turkish more than harmful Turkish. It reacts to the language, not the risk.
Fields: pair_id, category, lang, type (safe/harmful), label, prompt, flagged_by_<guard>.
Full method + a dependency-free refusal scorer for your own model: github.com/fevziegeyurtsevenler/turkish-over-refusal-set
Method: XSTest (arXiv:2308.01263) + OR-Bench (arXiv:2405.20947), extended to Turkish.
Related: guard-blindspots-tr · guardrail-arena · AltaySec
Apache-2.0 · by AltaySec.