dataset_info:
features:

• name: Prompt
  dtype: string
  splits:
• name: train
  num_bytes: 2366
  num_examples: 37
  download_size: 2228
  dataset_size: 2366
  configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*

Language Model Testing Dataset 📊🤖

Introduction 🌐

This repository provides a dataset inspired by the paper "Explore, Establish, Exploit: Red Teaming Language Models from Scratch" It's designed for anyone interested in testing language models (LMs) for biases, toxicity, and misinformation.

Dataset Origin 📝

The dataset is based on examples from Tables 7 and 8 of the paper, which illustrate how prompts can elicit not just biased but also toxic or nonsensical responses from LMs.

Toxicity and Untruths 🤬

The prompts here, derived from red-teaming GPT-3-text-davinci-002 with classifiers trained on the CREAK dataset, are intended to elicit responses that can reveal tendencies towards toxicity or untruths.

Nonsense Responses 🤪

Similarly, the prompts from Table 8 are structured to test LM responses for nonsensical or toxic content. These were initially used against GPT-3-text-davinci-002 with classifiers trained on ChatGPT-3.5-turbo labels.

Purpose of the Dataset 🎯

This dataset is provided as a tool for researchers and developers to test their own LMs. It's particularly useful for evaluating how different models handle potentially problematic content, whether it's biased, toxic, or factually incorrect.

Using the Dataset 🛠️

Feel free to use this dataset to assess the response patterns of any LM. It's a valuable resource for identifying areas where LMs might need improvement in handling sensitive or complex content.

Goal 🎖️

The aim is to facilitate broader research into making LMs safer, more reliable, and ethically responsible by providing a ready-to-use dataset for testing and analysis.