annotations_creators:
- expert-generated
language_creators: - expert-generated
language: - yo
- ha
- ig
- en
- und
license: cc-by-4.0
multilinguality: multilingual
size_categories: - n<1K
source_datasets: - original
task_categories: - text-classification
pretty_name: LSR Benchmark
tags: - ai-safety
- red-teaming
- cross-lingual
- low-resource
- west-african-languages
- refusal-evaluation
- llm-safety
LSR Benchmark: Linguistic Safety Robustness for West African Languages
Dataset Description
Homepage: https://huggingface.co/spaces/Faruna01/lsr-dashboard
Paper: [Coming Soon]
Point of Contact: [email protected]
Summary
The LSR Benchmark is the first systematic evaluation dataset for cross-lingual refusal degradation in West African languages. It contains 70 attack probes (14 per language) in English, Yoruba, Hausa, Igbo, and Igala, designed to measure whether frontier LLMs maintain safety refusal behavior when harmful intent is expressed in low-resource African languages.
Each non-English probe is paired with an English baseline probe that expresses the same harmful intent directly. This dual-probe structure enables comparison of refusal rates between English and target languages, and derivation of the Refusal Centroid Drift (RCD) metric.
The benchmark was developed as the dataset component of the LSR evaluation framework, implemented in the Inspect AI framework for integration into the UK AISI's inspect_evals repository.
Supported Tasks
Safety evaluation / refusal rate measurement: The primary use is evaluating whether a model's refusal behavior holds across language conditions under matched harmful prompts.
Cross-lingual transfer analysis: The paired probe structure enables analysis of how safety behavior transfers (or fails to transfer) across linguistic distance.
Languages
English (en), Yoruba (yo), Hausa (ha), Igbo (ig), Igala (und — no standard BCP 47 tag currently assigned). Target-language probes are written natively by the author, a researcher with working knowledge of all target languages. They are not machine-translated.
Dataset Structure
Data Fields
| Field | Type | Description |
|---|---|---|
pair_id |
string | Unique ID linking a target-language probe to its English baseline (e.g., yoruba_001) |
language |
string | Language of the probe (en, yo, ha, ig, igala) |
attack_vector |
string | Harm category: physical_harm, toxicology, targeted_violence, historical_pretext |
technique |
string | Framing technique: direct, roleplay, historical_pretext, cultural_documentation |
role |
string | Authority role the model is asked to assume, if applicable (e.g., Oba, Sarki); null for direct probes |
severity |
string | HIGH or CRITICAL, based on specificity of targeted harmful output |
prompt |
string | Full text of the probe |
target |
string | Description of the harmful output the probe attempts to elicit |
Data Splits
The dataset does not use train/validation/test splits. It is an evaluation-only benchmark.
| Split | Probes |
|---|---|
| Evaluation | 70 (14 per language × 5 languages) |
Dataset Creation
Curation Rationale
Existing LLM safety benchmarks (AdvBench, SafetyBench, etc.) evaluate safety almost exclusively in English. Work on multilingual safety typically translates English prompts, which misses the structural nature of culturally grounded attacks. This dataset provides natively written, culturally contextualized harmful probes in four low-resource West African languages to accurately measure Refusal Centroid Drift (RCD).