language:

• en
  license: odc-by
  task_categories:
• text-classification
  tags:
• safe
• safety
• jailbreak
• ai-safety
• llm
• lm
• moderation
• classification
• refusal
  extra_gated_prompt: >-
  Access to this dataset is automatically granted upon accepting the AI2
  Responsible Use Guidelines, and
  completing all fields below
  extra_gated_fields:
  Your full name: text
  Organization or entity you are affiliated with: text
  State or country you are located in: text
  Contact email: text
  Please describe your intended use of the low risk artifact(s): text
  I understand that this dataset is a research artifact that may contain or produce unfiltered, toxic, or harmful material: checkbox
  I agree to use this dataset for research purposes in accordance with the AI2 Responsible Use Guidelines: checkbox
  I agree that AI2 may use my information as described in the Privacy Policy: checkbox
  I certify that the information I have provided is true and accurate: checkbox
  configs:
• config_name: default
  data_files:
  • split: response_harmfulness
    path: data/response_harmfulness-*
  • split: response_refusal
    path: data/response_refusal-*
    dataset_info:
    features:
  • name: prompt
    dtype: string
  • name: response
    dtype: string
  • name: label
    dtype: string
  • name: prompt_type
    dtype: string
  • name: prompt_harm_category
    dtype: string
    splits:
  • name: response_harmfulness
    num_bytes: 427295
    num_examples: 446
  • name: response_refusal
    num_bytes: 430792
    num_examples: 449
    download_size: 431812
    dataset_size: 858087

Dataset Card for XSTest-Response

Disclaimer:

The data includes examples that might be disturbing, harmful or upsetting. It includes a range of harmful topics such as discriminatory language and discussions
about abuse, violence, self-harm, sexual content, misinformation among other high-risk categories. The main goal of this data is for advancing research in building safe LLMs.
It is recommended not to train a LLM exclusively on the harmful examples.

Dataset Summary

XSTest-Response is an artifact of WildGuard project, and the purpose of this dataset is to extend XSTest with model responses to directly evaluate moderator accuracy for scoring models on a real safety benchmark.

response_refusal split contains 449 prompts for refusal detection (178 refusals, 271 compliances).

response_harmfulness split contains 446 prompts for response harmfulness (368 harmful responses, 78 benign responses).

Please check the paper for further details on data construction: WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs.

Usage

from datasets import load_dataset

# Load the response_refusal split
dataset = load_dataset("allenai/xstest-response", split="response_refusal")

# Load the response_harmfulness split
dataset = load_dataset("allenai/xstest-response", split="response_harmfulness")

Dataset Details

The dataset contains the following columns:

• prompt: str, indicates the user request.
• response: str, or None for prompt-only items in WildGuardTrain.
• label: str, indicates the label of the prompt. It can be "refusal" or "compliance" for response_refusal split, and "harmful" or "unharmful" for response_harmfulness split.
• prompt_type: str ("prompt_harmful" or "prompt_safe"), indicates whether the prompt is harmful or safe.
• prompt_harm_category: str, indicates the XSTest category of the prompt. If contrast is included in the category, it means the prompt is generated to contrast with prompts in the same category, for example, figurative_language <-> contrast_figurative_language.

Citation

@misc{wildguard2024,
      title={WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs}, 
      author={Seungju Han and Kavel Rao and Allyson Ettinger and Liwei Jiang and Bill Yuchen Lin and Nathan Lambert and Yejin Choi and Nouha Dziri},
      year={2024},
      eprint={2406.18495},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2406.18495}, 
}