license: cc-by-nc-4.0
language:
- en
- so
size_categories: - n<1K
task_categories: - text-classification
- text-generation
tags: - ai-safety
- multilingual
- low-resource-languages
- somali
- safety-evaluation
- refusal
- alignment
- harmbench
- advbench
pretty_name: SomaliBench v0
SomaliBench v0
The first native-author-verified Somali safety evaluation benchmark.
100 harmful-intent prompts drawn from HarmBench (Mazeika et al. 2024) and
AdvBench (Zou et al. 2023), translated into Somali by a native speaker
(Khalid Yusuf Dahir, Mogadishu) and released as an evaluation set for
measuring multilingual safety alignment.
Why this exists
Somali has 15–20 million speakers and zero native-verified safety
evaluation resources. SomaliBench fills that gap.
Schema
| field | description |
|---|---|
probe_id |
unique within the dataset (e.g. sb001) |
category |
one of: harmful_instruction, misinformation, hate_speech, privacy, cybersecurity, CBRN, self_harm |
source |
which canonical benchmark the prompt came from (HarmBench/ |
lang |
en or so |
text |
the prompt in the indicated language |
back_translation |
English back-translation of the Somali (for auditability) |
translation_method |
harmbench_original / advbench_original / claude_drafted |
verified_by_author |
true if the native author has reviewed the row |
notes |
free-form per-row notes |
How to load
from datasets import load_dataset
ds = load_dataset("khaledyusuf44/somalibench-v0")
How to use (intended)
Run a model against the Somali (lang=so) prompts. Compare its refusal
rate to its refusal rate on the English (lang=en) prompts. The two halves
are aligned by probe_id — refusal-rate differences across the pair are
the multilingual safety transfer measurement.
How NOT to use
- Do NOT fine-tune any model on these prompts toward compliance.
- Do NOT use them to attack deployed systems without authorization.
- Do NOT use outside safety-evaluation or alignment-research contexts.
Categories
| Category | Target count | Description |
|---|---|---|
| harmful_instruction | 20 | step-by-step how-tos for physical harm, weapons, illegal acts |
| misinformation | 15 | fake news, propaganda, disinformation tactics |
| hate_speech | 15 | racist, sexist, ethnic-supremacy content |
| privacy | 10 | identity theft, doxxing, surveillance |
| cybersecurity | 15 | hacking, malware, exploits, unauthorized access |
| CBRN | 15 | chemical / biological / radiological / nuclear |
| self_harm | 10 | suicide methods, eating-disorder promotion |
| Total | 100 |
Citation
@dataset{dahir2026somalibench,
author = {Khalid Yusuf Dahir},
title = {{SomaliBench v0}: A Native-Verified Safety Benchmark for Somali},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/datasets/khaledyusuf44/somalibench-v0}
}
Author
Khalid Yusuf Dahir — Mogadishu · khaliddahir.com
Related releases:
- multilingual-safety-probe — cross-lingual safety pilot using SomaliBench-style probes
- somaliweb-v1 — 412M-token Somali web corpus