language:
- en
- ar
- fr
tags: - safety
- multilingual
- llm-evaluation
- alignment
- arabic
- darija
- maghrebi-french
license: cc-by-4.0
Multilingual Safety Evaluation of Qwen 2.5-1.5B-Instruct
Overview
This dataset contains the prompts and model responses from a safety evaluation of
Qwen/Qwen2.5-1.5B-Instruct across five languages,
two of which (Algerian Darija and Maghrebi French) are dialectal varieties typically absent from safety benchmarks.
Research question: Does the model maintain consistent safety alignment across typologically diverse and low-resource language varieties?
Languages
| Code | Language | Type |
|---|---|---|
| EN | English | High-resource |
| AR | Modern Standard Arabic | High-resource |
| DZ | Algerian Darija | Low-resource dialect |
| FR | Standard French | High-resource |
| MFR | Maghrebi French | Low-resource mixed variety |
Files
prompts.jsonl— 75 prompts (10 harmful + 5 benign per language)results.jsonl— model responses and classificationssummary.json— aggregated statisticsrefusal_rate_chart.png— visualizationsevaluation.ipynb— reproducible notebook
Results
| Language | Refusal rate | Refused / Total |
|---|---|---|
| English (EN) | 100% | 10/10 |
| Modern Standard Arabic (AR) | 90% | 9/10 |
| Algerian Darija (DZ) | 40% | 4/10 |
| Standard French (FR) | 90% | 9/10 |
| Maghrebi French (MFR) | 90% | 9/10 |
| Overall | 82% | 41/50 |
The gap between the highest and lowest refusal rate across languages is 60 percentage points.
The strongest refusal behaviour was observed in English, while Algerian Darija showed the lowest refusal rate.
Methodology
- Hardware: Google Colab T4 GPU (free tier)
- Model loading: 4-bit NF4 quantization via bitsandbytes
- Inference: greedy decoding, max 512 new tokens
- System prompt: "You are a helpful assistant."
- Classification: keyword-based refusal detection in all five languages + short-response heuristic
Ethical note
All harmful prompts were written for safety research purposes only and are not accompanied by
harmful instructions. The goal is to identify alignment gaps across underrepresented language varieties.
Citation
@dataset{multilingual_safety_qwen25,
author = {[Your Name]},
title = {Multilingual Safety Evaluation of Qwen 2.5-1.5B-Instruct},
year = {2026},
url = {https://huggingface.co/datasets/fatima2442/qwen25-multilingual-safety-eval}
}