language:

• tr
  license: other
  license_name: mixed-cc0-cc-by-4.0-apache-2.0
  license_link: LICENSE.md
  pretty_name: Turkish Prompt Injection Benchmark
  task_categories:
• text-classification
  size_categories:
• 1K<n<10K
  tags:
• prompt-injection
• jailbreak-detection
• llm-security
• guardrails
• benchmark
• turkish
  configs:
• config_name: full
  default: true
  data_files:
  • split: test
    path: data/full.jsonl
• config_name: attacks
  data_files:
  • split: test
    path: data/attacks.jsonl
• config_name: false_alarm
  data_files:
  • split: test
    path: data/false_alarm.jsonl

Turkish Prompt Injection Benchmark

One place to test a prompt-injection guard on Turkish text from both sides: does it catch attacks, and does it leave ordinary Turkish alone?

3,372 rows: 317 attacks from five public Turkish sets, and 3,055 benign rows. The benign rows are the hard negatives that ship with those sets, plus 2,875 everyday business texts and AI task instructions.

No new text was written for this benchmark. Every row comes from a published dataset and keeps its source, its row id there, and its licence.

Results

Each attack column shows recall: the share of that subset's attacks the guard catches. False alarms are counted over all 3,055 benign rows. I ran every model with the same code, using each model's own injection label. These are my measurements, not the authors' published numbers.

How to read it:

• Look at recall and false alarms together. A guard that flags everything has perfect recall, and one that flags nothing has no false alarms. F1 over all rows is dominated by the benign rows, because there are ten times more of them, so it mostly rewards few false alarms.
• No model wins on both. proventra catches the most attacks on average and blocks 27% of ordinary text, including 77% of the task instructions. My models block 1–2% and catch fewer attacks from sources they have not seen. v2 uses its recommended threshold 0.05 (v2.1); at its first threshold 0.3 the row read 0.333 / 0.924 / 0.288 / 0.944 / 0.952, mean 0.688, 26 false alarms.
• * berat is a Turkish translation of deepset/prompt-injections. A model trained on the English deepset set has, in effect, seen these items.
• ‡ My own models. conv, 1k and berat each come from a model trained without that source; fevzi, invisible and the false alarms come from the released model, which never saw them. My models were trained on other texts from the same three domains as the false-alarm rows (business, banking, PII instructions), so their false-alarm numbers are in-domain and the others' are not.

The released models trained on all three sources score higher on conv, 1k and berat (for example v2: 0.900, 1.000, 0.932), but those are the same authors and style as their training data. I leave those numbers out of this table.

Subsets

Columns: id, text, label (1 = injection, 0 = benign), subset, category (attack family or benign category from the source, where it has one), source, source_id, license.

Changes from the sources: in fevzi and invisible the placeholders {SECRET}, {URL} and {EMAIL} are replaced with the fixed fake values sk-test-4f2a, ornek-log.net and [email protected]. In task_instruction the instruction and the input are joined with a blank line. Nothing else is changed. Nine texts appear twice in berat because they appear twice in its source test split.

How to evaluate

python evaluate.py --model halilneed/turkish-prompt-injection-classifier --threshold 0.3
python evaluate.py --model protectai/deberta-v3-base-prompt-injection-v2 --config attacks

evaluate.py is in this repository. It prints recall and false-positive rate per subset, the mean attack recall, the false alarms and F1 over all rows.

If you train on any of the source datasets, do not report this benchmark as unseen for those subsets. Use their train splits only and say so, as in the ‡ rows.

Limits

• Small attack subsets. 30, 79, 59, 107 and 42 attacks. One or two rows move recall by several points.
• Synthetic or translated. None of the rows is real production traffic.
• Different definitions. The source sets agree on the core, which is overriding instructions, leaking the system prompt, abusing tools and exfiltrating data. At the edges they differ: the deepset translation counts some role-play requests as injection.
• Single texts. No conversation history, no system prompt, no tool context.

Türkçe

Bir prompt injection koruma modelini Türkçe metinde iki yönden test etmek için tek set: saldırıları yakalıyor mu, sıradan Türkçeyi rahat bırakıyor mu?

• 3.372 satır: beş açık Türkçe setten 317 saldırı, 3.055 zararsız satır. Zararsız satırlar, o setlerin kendi zor örnekleri ile 2.875 sıradan iş metni ve görev talimatından oluşuyor.
• Üç yapılandırma: full (hepsi), attacks (açık setlerin test bölmeleri), false_alarm (yalnızca zararsız).
• Yeni metin yazılmadı. Her satır kaynağını, kaynaktaki kimliğini ve lisansını taşıyor.
• Hiçbir model iki yönde birden önde değil. proventra ortalamada en çok saldırıyı yakalıyor (0.865), ama sıradan metnin %27'sini engelliyor. Benim modellerim %1–2 engelliyor ama görmediği kaynaklardan daha az saldırı yakalıyor (v2, 0,05 eşikte: 0.741).
• Kaynak setlerden biriyle eğittiyseniz o alt seti "görülmemiş" diye raporlamayın.

Licence and attribution

This collection is released under mixed licences; each row's license column applies to that row. See LICENSE.md.

• CC-BY-4.0: conv and 1k by Enes Deniz (3nesdeniz); fevzi and invisible by fevziegeyurtsevenler. Changes are listed above.
• Apache-2.0: berat by beratcmn, translated from deepset/prompt-injections (Apache-2.0).
• CC0: personal_data_business and banking by halilneed; task_instruction from cagrigungor.

Thank you to the authors for publishing them.

The model measured on this set: halilneed.agency/models/turkish-prompt-injection-classifier · All sets and models: halilneed.agency/models