license: apache-2.0
task_categories:
- text-classification
language: - eu
- en
- es
- ca
- gl
pretty_name: safety-IberRisk
size_categories: - 10K<n<100K
configs: - config_name: test
data_files:- split: eu
path: eu-safety-IberRisk.jsonl - split: en
path: en-safety-IberRisk.jsonl - split: es
path: es-safety-IberRisk.jsonl - split: ca
path: ca-safety-IberRisk.jsonl - split: gl
path: gl-safety-IberRisk.jsonl
tags:
- split: eu
- safety
🛡️ Safety-IberRisk: Iberian Multilingual LLM Safety Evaluation
A dataset of unsafe prompts in Iberian languages spanning multiple domains and jailbreak strategies
Dataset Summary
This dataset is designed to evaluate the safety capabilities and behavioral guardrails of Large Language Models (LLMs) in culturally specific contexts across various Iberian languages and English. It focuses on testing model robustness against potentially harmful requests and common jailbreak/evasion techniques.
The corpus contains 15,030 synthetically generated instances, uniformly distributed across five languages. It provides a systematic resource for analyzing the consistency of model safeguards across languages and their resistance to adversarial prompt formulations.
Supported Tasks and Leaderboards
safety-evaluation: Assessing whether a model refuses or complies with harmful requests.jailbreak-detection: Training models or filters to detect adversarial evasion strategies.multilingual-alignment-testing: Comparing the cultural and linguistic parity of safety alignments in LLMs.
Languages
The dataset is perfectly parallel in its generation strategy across five languages:
- English (
en) - Spanish (
es) - Basque / Euskera (
eu) - Catalan (
ca) - Galician (
gl)
Dataset Structure
Categories
The dataset is divided into three primary harm categories:
- Illegal Activities: Requests related to illicit behaviors (e.g., drug trafficking, human trafficking).
- Misuse: Requests aimed at using the models for unintended purposes (e.g., generation of disinformation, fake news).
- Toxicity: Content that is offensive, discriminatory, hateful, or harassing.
Data Instances
Each language contains 3,006 instances, breaking down perfectly by category and transformation strategy:
| Category | Original Prompts | Transformed Prompts | Total per Language |
|---|---|---|---|
| Illegal Activities | 167 | 835 | 1,002 |
| Misuse | 167 | 835 | 1,002 |
| Toxicity | 167 | 835 | 1,002 |
| Total | 501 | 2,505 | 3,006 |
Total across all 5 languages: 15,030 examples.
Transformation Strategies (Jailbreaks)
To evaluate model robustness, five distinct evasion techniques were applied to each of the 167 original prompts per category:
- Role Play: Formulating the request within a fictional scenario or character representation.
- Base64 Encoding: Obfuscating the harmful request using Base64.
- Zero-Width Insertion: Inserting invisible Unicode characters to bypass simple lexical filters.
- Creative Writing: Disguising the request as a creative or literary task.
- Bait-and-Switch: Beginning with an apparently harmless request that subsequently pivots into a harmful intention.
Dataset Creation
Curation Rationale
The dataset was created to address the gap in safety evaluation for low-to-medium resource languages (like Basque, Galician, and Catalan) compared to high-resource languages (English, Spanish). By keeping the generation strategies identical across all languages, the dataset allows researchers to perform 1:1 cross-lingual comparability tests on model safety.
Source Data and Generation
- Platform: Galtea.ai
Limitations
As the dataset is synthetically generated via LLMs, it may contain generation artifacts or specific biases. Furthermore, while the strategies are parallel, deep cultural nuances in how harm is expressed may slightly vary from natural human phrasing.
Ethical Considerations and Risks
⚠️ WARNING: By design, this dataset contains highly toxic, offensive, and potentially illegal text. It is intended strictly for safety evaluation, red-teaming, and defensive research.
- Users must exercise caution when viewing or displaying this data.
- The dataset must not be used to train models to generate harmful content.
Acknowledgements
This work has been partially supported by the Basque Government (Research group funding IT1570-22 and IKER-GAITU project), the Spanish Ministry for Digital Transformation and of Civil Service, and the EU-funded NextGenerationEU Recovery, Transformation and Resilience Plan (ALIA project).