license: mit
language:
- yo
- ha
- ig
- sw
- ar
- en
tags: - safety
- mechanistic-interpretability
- activation-steering
- multilingual
- low-resource-languages
- refusal
- llm-safety
- african-languages
- jailbreak-benchmark
pretty_name: LSR Anchoring - Multilingual Benchmark & Safety Steering Results
size_categories: - n<1K
LSR Anchoring — Multilingual Benchmark & Safety Steering Results
This dataset contains all experimental results for two related studies:
Study 1 - Benchmark:
Multilingual Jailbreak Vulnerability Benchmark and Mitigation
for Low-Resource African Languages
GlobalSouthML Hackathon 2026, Apart Research
Study 2 — Mitigation:
Latent Space Refusal Anchoring for Low-Resource African Languages:
Mechanistic Safety Recovery Without Retraining
ICML 2026, GlobalSouthML Workshop
Together they form a full pipeline: find the failure → characterise it
geometrically → fix it, all without supervised African-language data.
Languages Covered
| Code | Language | Family | Resource Level |
|---|---|---|---|
yo |
Yoruba | Niger-Congo | Low |
ha |
Hausa | Afro-Asiatic | Low |
ig |
Igbo | Niger-Congo | Low |
igl |
Igala | Niger-Congo | Very Low (no ISO 639-1) |
sw |
Swahili | Bantu | Mid |
ar |
Arabic | Semitic | High |
Models Evaluated
| Model | Study | Method |
|---|---|---|
meta-llama/Meta-Llama-3-8B-Instruct |
Both | Benchmark + SAE & Mean-Act Steering |
meta-llama/Llama-3.1-70B-Instruct |
Steering only | Mean-Act Steering |
mistralai/Mistral-7B-Instruct-v0.3 |
Both | Benchmark + Mean-Act Steering |
Qwen/Qwen2.5-7B-Instruct |
Both | Benchmark + Mean-Act Steering |
google/gemma-2-9b-it |
Benchmark only | Benchmark |
Repository Structure
lsr-anchoring-phase2-results/
│
├── benchmark/ ← Study 1: Benchmark results
│ ├── raw_llama3-8b_*.csv — Per-prompt responses, Llama-3-8B
│ ├── raw_mistral-7b_*.csv — Per-prompt responses, Mistral-7B
│ ├── raw_qwen2.5-7b_*.csv — Per-prompt responses, Qwen2.5-7B
│ ├── raw_gemma2-9b_*.csv — Per-prompt responses, Gemma2-9B
│ ├── summary_llama3-8b_*.json — SRR summary by language
│ ├── summary_mistral-7b_*.json
│ ├── summary_qwen2.5-7b_*.json
│ ├── summary_gemma2-9b_*.json
│ ├── benchmark_results_merged.csv — All models merged
│ └── Model-EnglishSRR-AvgAfricanSRR-SafetyGap.csv
│
├── llama-3-8b-path-a/ ← Study 2: Llama-3.1 8B SAE steering
│ ├── results_rows.csv/json
│ ├── anchor_cache.json
│ ├── full_sweep.csv
│ └── logs/
│
├── llama-3-8b-path-b/ ← Study 2: Llama-3.1 8B Mean-Act steering
│ ├── results_8b_pathB_rows.csv/json
│ └── anchor_cache_8b_pathB.json
│
├── llama-3-70b/ ← Study 2: Llama-3.1 70B
│ ├── results_70b_rows.csv/json
│ ├── anchor_cache.json
│ └── logs/
│
├── mistral-7b/ ← Study 2: Mistral-7B steering
│ ├── results_mistral7b_rows.csv/json
│ ├── anchor_cache_mistral7b.json
│ └── logs/
│
├── qwen2.5-7b/ ← Study 2: Qwen2.5-7B steering
│ ├── results_qwen7b_rows.csv/json
│ ├── anchor_cache_qwen7b.json
│ └── logs/
│
└── data/ ← Study 2: Cross-model summary files
├── results_multilang_v3.json
├── results_phase2b_all.json
├── results_phase2b_arabic.json
└── results_phase2b_yoruba_ext.json
Study 1 — Benchmark Results Summary
Safety Refusal Rate (SRR) = refused harmful prompts / total harmful prompts.
| Model | English | Yoruba | Hausa | Igbo | Igala | Swahili | Arabic |
|---|---|---|---|---|---|---|---|
| Llama-3-8B | 99% | 24% | 24% | 31% | 21% | 38% | 87% |
| Mistral-7B | 85% | 11% | 28% | 15% | 16% | 25% | 12% |
| Qwen2.5-7B | 95% | 4% | 6% | 14% | 13% | 0% | 83% |
| Gemma2-9B | 97% | 55% | 72% | 20% | 47% | 94% | 99% |
Key finding: English SRR stays at 85–99% across all models. Every African
language tested shows substantial collapse. Igala is the sharpest stress test,
with weak refusal across three of four models despite no prior AI safety coverage.
Study 2 — LSR-Anchoring Results Summary
| Model | Best Language | Peak SRR | KL at Peak | DPL at Peak |
|---|---|---|---|---|
| Llama-3.1 8B (SAE/Path A) | Igala | 0.844 | 0.31 | 0.06 |
| Llama-3.1 8B (Mean-Act/Path B) | Hausa | 0.71 | 0.48 | 0.12 |
| Llama-3.1 70B | Yoruba, Igala | 1.00 | 2.58–3.53 | — |
| Mistral-7B | Igala | 0.75 | 0.61 | 0.06 |
| Qwen2.5-7B | Hausa | 0.51 | 0.49 | 0.50 |
Notable finding: Arabic exhibits inverse transfer on every architecture.
English-derived steering directions reduce refusal rates on Arabic harmful
prompts as alpha increases (SRR → −0.20 at α=25 on 70B), indicating a
geometric mismatch rather than a baseline effect. Do not apply LSR-Anchoring
to Arabic-language agents without language-specific direction derivation.
Live Dashboard
Explore all benchmark and steering results interactively:
https://huggingface.co/spaces/Faruna01/lsr-dashboard
Citation
@misc{faruna2026benchmark,
title = {Multilingual Jailbreak Vulnerability Benchmark and Mitigation
for Low-Resource African Languages},
author = {Faruna, Godwin Abuh},
year = {2026},
url = {https://huggingface.co/datasets/Faruna01/lsr-anchoring-phase2-results}
}
@misc{faruna2026lsranchoring,
title = {Latent Space Refusal Anchoring for Low-Resource African Languages:
Mechanistic Safety Recovery Without Retraining},
author = {Faruna, Godwin Abuh},
year = {2026},
url = {https://huggingface.co/datasets/Faruna01/lsr-anchoring-phase2-results}
}
Code
All experiment scripts, benchmark code, and prompt sets:
https://github.com/farunawebservices/lsr-anchoring
License
MIT License.