license: cc-by-sa-4.0
language:
- en
- ru
- sr
- te
task_categories: - text-classification
tags: - safety
- harm-detection
- multilingual
- cross-lingual
- evaluation
size_categories: - n<1K
configs: - config_name: default
data_files:- split: test
path: data/multilingual_test.parquet
- split: test
Multilingual safety-probe test set (EN / RU / SR-Latn / SR-Cyrl / TE)
⚠️ Content warning. Half of the items are harmful prompts (violence and
fraud requests, e.g. instructions for hurting people or scamming them), in
five language versions. They are released for safety-evaluation research
only. Reading them may be disturbing or upsetting.
Contents
1,000 rows = 200 curated English items × 5 language versions
(English original + Russian, Serbian-Latin, Serbian-Cyrillic, Telugu
translations). The 200 items are 100 harmful (50 violence / 50 fraud) and 100
benign (35 violence-adjacent near-miss, 35 fraud-adjacent near-miss, 30 clearly
benign). Files: data/multilingual_test.parquet and an identicaldata/multilingual_test.csv.
| column | type | meaning |
|---|---|---|
item_id |
string | stable id; prefix identifies the upstream dataset |
language |
string | en, ru, sr, te |
script |
string | en_Latn, ru_Cyrl, sr_Latn, sr_Cyrl, te_Telu |
text |
string | the prompt (final validator-corrected text for translations) |
label |
int | 1 = harmful, 0 = benign |
upstream_source |
string | dataset the English item was selected from |
upstream_license |
string | license of that upstream dataset (see mix below) |
validated |
bool | row passed human review (see Validation; False for exactly 1 Telugu row) |
correction_applied |
bool | a human validator's correction replaced the machine translation (87 rows) |
Provenance
| upstream dataset | items (of 200) | rows (of 1,000) | license |
|---|---|---|---|
| nvidia/Aegis-AI-Content-Safety-Dataset-2.0 | 82 harmful | 410 | CC-BY-4.0 |
| bench-llm/or-bench (or-bench-80k) | 58 benign | 290 | CC-BY-4.0 |
| databricks/databricks-dolly-15k | 30 benign | 150 | CC-BY-SA-3.0 |
| XSTest | 12 benign | 60 | CC-BY-4.0 |
| StrongREJECT | 10 harmful | 50 | MIT |
| JailbreakBench/JBB-Behaviors | 5 harmful | 25 | MIT |
| HarmBench | 3 harmful | 15 | MIT |
All 10 StrongREJECT items trace to MIT-licensed provenance within StrongREJECT
(8 custom, 2 DAN); none come from its unlicensed upstream sources. English
items were selected for unambiguous labels, translatability (no wordplay or
culture-bound phrasing), and were decontaminated against WildGuardMix (the
probe training source).
How translations were produced and validated
- Machine translation:
claude-opus-4-8(adaptive thinking), one language
at a time, with LaBSE cross-lingual similarity gating: acceptance floor 0.75,
up to 5 attempts per item; 100% of released rows scored ≥ 0.75. - Serbian scripts: translated into Latin script, then transliterated to
Cyrillic withcyrtranslitunder a Latin→Cyrillic→Latin round-trip check;
ambiguous nj/lj/dž digraphs were individually human-confirmed before any
Cyrillic correction entered the dataset. - Human validation: two native/proficient validators per language, each
judging 150 items on three checks (meaning preserved / intensity preserved /
reads naturally) with a 100-item overlap for agreement. Disagreements were
resolved accept-or-correct with a deterministic tiebreak. Inter-annotator
agreement (Gwet's AC1) on the overlap: Russian 0.785, Serbian 0.870,
Telugu 0.855. - Corrections: 87 of 800 translated rows carry a validator correction
(correction_applied = True);textis always the final corrected version. - Coverage: every translated row was human-reviewed except one Telugu row
(aegis_f71a0126043c, left blank by its assigned validator), markedvalidated = False. English rows are the curated source items
(validated = Truerefers to the curation review, not translation review).
Intended use
Evaluating cross-lingual generalization of safety classifiers and
activation-space harm probes — in particular recall gaps between English and
lower-resource languages/scripts, and threshold-calibration effects. Not a
training set; not a jailbreak benchmark (all prompts are vanilla, direct
requests).
Licensing and attribution
The whole set is released under CC-BY-SA-4.0
(https://creativecommons.org/licenses/by-sa/4.0/). Share-alike is required by
the 30 Dolly-derived items (CC-BY-SA-3.0, upgraded per its §4(b) compatibility
path) and is compatible with the CC-BY-4.0 and MIT upstream sources. Theupstream_license column preserves the per-item mix; MIT copyright notices for
StrongREJECT, JailbreakBench, and HarmBench are in NOTICE. Translations are
adaptations of the upstream English prompts; modifications (translation +
validator corrections) are indicated per row.
Attribution: NVIDIA (Aegis 2.0, itself derived from Anthropic hh-rlhf, MIT),
OR-Bench (Cui et al.), Databricks (dolly-15k; Wikipedia-derived passages also
credit Wikipedia contributors, CC-BY-SA), XSTest (Röttger et al.), StrongREJECT
(Souly et al.), JailbreakBench (Chao et al.), HarmBench (Mazeika et al.).