license: apache-2.0
task_categories:
- text-classification
language: - eu
pretty_name: safety-GuardEUS
size_categories: - n<1K
configs: - config_name: test
data_files:- split: eu
path: safety-GuardEUS.jsonl
- split: eu
🛡️ Safety-GuardEUS
An evaluation benchmark to test AI safety filters and guardrail models in the Basque language.
Dataset Summary
Safety-GuardEUS is a specialized evaluation dataset designed to test and benchmark the performance of AI guardrail models, moderation classifiers, and LLM safety filters in the Basque language (Euskara).
As language models become more prevalent, ensuring they operate safely across all languages is critical. This dataset provides a rigorous, localized benchmark to evaluate how well safety systems can distinguish between safe, helpful responses and unsafe, policy-violating responses in Basque.
Dataset Structure
The dataset consists of exactly 250 evaluation pairs, constructed symmetrically to test a model's ability to differentiate between safe and unsafe outputs for the exact same context.
- Prompts: 25 unique user prompts (questions/requests) distributed across 5 safety categories.
- Responses: Each of the 25 prompts is paired with 10 distinct responses:
- 5 Safe Answers: Helpful, policy-compliant, or appropriate refusals.
- 5 Unsafe Answers: Policy-violating responses that models should flag or block.
- Total: 25 prompts × 10 responses = 250 evaluation rows.
Categories
The 25 prompts are divided equally among the following 5 critical safety risk categories:
self-harm: Content encouraging, providing instructions for, or glorifying self-injury or suicide.drugs: Content involving the manufacture, distribution, or promotion of illegal substances.child-exploitation: Content that endangers minors or violates child safety policies.terrorism: Content promoting terrorism, violent extremism, or providing instructions for mass harm.explicit-content: Non-consensual, highly explicit, or otherwise prohibited sexual content.
Data Fields
| Field | Type | Description |
|---|---|---|
question |
string | The user's input/request in Basque. |
answer |
string | The system's response in Basque. |
label |
string | safe or unsafe |
category |
string | Harm category (e.g., self-harm, drugs, child-exploitation...) |
Ethical Considerations and Risks
⚠️ WARNING: By design, this dataset contains highly toxic, offensive, and potentially illegal text. It is intended strictly for safety evaluation, red-teaming, and defensive research.
- Users must exercise caution when viewing or displaying this data.
- The dataset must not be used to train models to generate harmful content.
Acknowledgements
This work has been partially supported by the Basque Government (Research group funding IT1570-22 and IKER-GAITU project), the Spanish Ministry for Digital Transformation and of Civil Service, and the EU-funded NextGenerationEU Recovery, Transformation and Resilience Plan (ALIA project).