license: cc-by-4.0
task_categories:

• text-classification
• text-generation
• text2text-generation
  language:
• en
  pretty_name: RealHarm
  size_categories:
• n<1K
  tags:
• guardrail-evaluation
  configs:
• config_name: realharm
  data_files:
  • split: safe
    path: data/realharm_safe.jsonl
  • split: unsafe
    path: data/realharm_unsafe.jsonl

RealHarm

RealHarm is a collection of harmful real-world interactions with AI agents.

Dataset Details

Dataset Description

RealHarm contains harmful samples, categorized among 10 harm categories. A complete taxonomy has been proposed along with the dataset and is described in the RealHarm paper. Each sample has an associated safe version, for which we rewrote the agent answer to make it harmless.

This dataset provides researchers and developers with authentic examples of AI systems producing dangerous, offensive, or inappropriate content. RealHarm offers crucial insights into the challenges and risks of deploying conversational AI in real-world scenarios. This resource can be instrumental in improving AI safety measures, developing more robust content filtering systems, and enhancing the overall reliability of AI agents.

• Curated by: Matteo Dora, Pierre Le Jeune, Luca Rossi
• Language(s) (NLP): English, German, Russian, Korean, Finnish
• License: CC BY 4.0

Dataset Sources

The samples were collected mostly from the AI Incident Database, and other online sources.

• Website: Giskard
• Repository: https://github.com/Giskard-AI/realharm
• Paper: Arxiv

Uses

Direct Use

This dataset should be used to assess

Dataset Structure

The dataset is split into two splits: safe and unsafe containing the same number of samples. Each sample contains the following fields:

context: # a context sentence describing the chatbot and its use case
source: # an URL to the original reporting of the 
language: # the language code of the conversation (ISO 639-1 format)
taxonomy: # list of categories of harms 
label: # safe / unsafe
conversation: # the complete interaction with the AI agent as a list of messages
- role: user
  content: 
- role: agent
  content: 

Dataset Creation

The data collection focused on failure examples from AI systems, particularly language models with text-to-text interfaces such as chatbots and question-answering systems. The scope was limited to documented interactions showing credible evidence of failures, with genuine harms like reputation damage, legal liability, and financial loss. Interactions had to be verifiable through sources like reliable news organizations or official statements. For each failure, the interactions were rewritten to remove the misbehavior using GPT-4o and were manually reviewed to ensure quality.

Bias, Risks, and Limitations

RealHarm limitations: (1) it focuses only on text-to-text AI systems, excluding multimodal systems that may present different risk profiles; (2) it relies on publicly reported incidents, introducing selection bias as private incidents remain invisible to our analysis; and (3) the final dataset contains only 68 fully documented examples despite reviewing hundreds of potential incidents, limiting statistical generalizability while still offering valuable qualitative insights for risk assessment.

Dataset Card Authors

• Pierre Le Jeune ([email protected])
• Jiaen Liu
• Luca Rossi
• Matteo Dora ([email protected])