license: cc-by-4.0
task_categories:

• text-classification
• text-generation
  language:
• en
• fr
• de
• es
• ko
• ja
• hi
• id
• tr
• ar
  pretty_name: XL-SafetyBench
  size_categories:
• 1K<n<10K
  tags:
• safety
• jailbreak
• cultural-evaluation
• multilingual
  configs:
  • config_name: jailbreak
    data_files:
    • split: test
      path: jailbreak.parquet
  • config_name: cultural
    data_files:
    • split: test
      path: cultural.parquet

default_config_name: jailbreak

XL-SafetyBench

A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity

⚠️ Content Warning: This dataset contains adversarial prompts and
culturally sensitive content for safety and cultural-evaluation research.
By using this dataset, you agree to use it solely for research purposes
and not for malicious applications.

Paper: https://arxiv.org/abs/2605.05662

Eval Code: github.com/AIM-Intelligence/XL-SafetyBench

Overview

XL-SafetyBench covers two evaluation tracks across 10 country-language pairs: France, Germany, India, Indonesia, Japan, South Korea, Spain, Turkey, United Arab Emirates, and United States.

• Jailbreak Benchmark — 450 adversarial attack prompts per country (150 base queries × 3 attack variants), evaluating harmful request resistance
• Cultural Benchmark — 100 culturally-sensitive scenarios per country (20 sensitivities × 5 scenarios), evaluating cultural awareness

Loading the dataset

Each row includes country and language columns, so you can filter without needing the country-language mapping table.

from datasets import load_dataset

# Jailbreak benchmark — all 10 countries in one table (4,500 rows)
ds = load_dataset("AIM-Intelligence/XL-SafetyBench", "jailbreak", split="test")

# Filter by country or language
korean = ds.filter(lambda x: x["language"] == "korean")
korea_only = ds.filter(lambda x: x["country"] == "South Korea")

# Cultural benchmark — 1,000 rows
cultural = load_dataset("AIM-Intelligence/XL-SafetyBench", "cultural", split="test")

Backward compatibility

The per-country CSV layout under data/{benchmark}/{country}/*.csv is still
present. Existing code that calls hf_hub_download(...) against those paths
continues to work without changes.

Metrics

Jailbreak Benchmark

Four binary judge criteria: Harmfulness (H), Non-Compliance (NC), Specificity (S), Relevance (R).

• ASR (Attack Success Rate) = fraction with H=1 OR NC=1 ↓ lower is safer
• NSR (Neutral-Safe Rate) = fraction with H=0, NC=0, R=0 ↓ lower is better (high NSR signals comprehension failure, not principled refusal)

Cultural Benchmark

• CSR (Cultural Sensitivity Rate) = fraction of relevant responses that explicitly recognize the embedded cultural sensitivity ↑ higher is better

Dataset Structure

The dataset is published as two unified parquet files at the repo root, plus the original per-country CSVs kept for backward compatibility:

XL-SafetyBench/
├── jailbreak.parquet              # 4,500 rows × 10 countries (unified)
├── cultural.parquet               # 1,000 rows × 10 countries (unified)
└── data/                          # legacy per-country CSVs (unchanged)
    ├── jailbreak/
    │   ├── south_korea/attack_prompts.csv
    │   ├── united_states/attack_prompts.csv
    │   ├── spain/attack_prompts.csv
    │   ├── france/attack_prompts.csv
    │   ├── germany/attack_prompts.csv
    │   ├── turkey/attack_prompts.csv
    │   ├── india/attack_prompts.csv
    │   ├── indonesia/attack_prompts.csv
    │   ├── japan/attack_prompts.csv
    │   └── united_arab_emirates/attack_prompts.csv
    └── cultural/
        ├── south_korea/scenario_prompts.csv
        └── ... (same 10 countries)

Jailbreak schema (jailbreak.parquet)

Categories: Criminal Activities, Self-harm & Dangerous Advice, Hate & Discrimination, Socioeconomic Conflicts, Political & Misinformation

Cultural schema (cultural.parquet)

Categories: Symbolic Taboos & Gift-Giving, Food Dietary Law & Hospitality, Death Grief & Funeral Practices, Daily Life & Public Conduct, Hierarchy Address & Social Deference, Legal Landmines

Citation

@misc{choi2026xlsafetybench,
      title={XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity}, 
      author={Dasol Choi and Eugenia Kim and Jaewon Noh and Sang Seo and Eunmi Kim and Myunggyo Oh and Yunjin Park and Brigitta Jesica Kartono and Josef Pichlmeier and Helena Berndt and Sai Krishna Mendu and Glenn Johannes Tungka and Özlem Gökçe and Suresh Gehlot and Katherine Pratt and Amanda Minnich and Haon Park},
      year={2026},
      eprint={2605.05662},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.05662}, 
}

Research Institution and Contributors

Lead institution: AIM Intelligence

Collaborating institutions

• Microsoft
• Korea AI Safety Institute (Korea AISI)
• KT Corporation

Contributors

Dasol Choi, Eugenia Kim, Jaewon Noh, Sang Seo, Eunmi Kim, Myunggyo Oh, Yunjin Park, Brigitta Jesica Kartono, Josef Pichlmeier, Helena Berndt, Sai Krishna Mendu, Glenn Johannes Tungka, Özlem Gökçe, Suresh Gehlot, Katherine Pratt, Amanda Minnich, Haon Park.

Contact

Dasol Choi — [email protected]