license: mit
language: en
multilinguality: monolingual
source_datasets: extended|Anthropic/hh-rlhf
task_categories:

• text-generation
• text2text-generation
  size_categories:
• 1K<n<10K
  tags:
• safety
• harm
• LLMs
• Vulnerability
• red teaming
• toxicity

AttaQ Dataset Card

The AttaQ red teaming dataset, consisting of 1402 carefully crafted adversarial questions, is designed to evaluate Large Language Models (LLMs) by assessing their tendency to generate harmful or undesirable responses.
It may serve as a benchmark to assess the potential harm of responses produced by LLMs.
The dataset is categorized into seven distinct classes of questions: deception, discrimination, harmful information, substance abuse, sexual content, personally identifiable information (PII), and violence.
Researchers and developers can use this dataset to assess the behavior of LLMs and explore the various factors that influence their responses, ultimately aiming to enhance their harmlessness and ethical usage.

The development of AttaQ involved multiple sources:

1. Extraction: Initially, attacks were methodologically extracted from the hh-rlhf dataset provided by Anthropic. These extracted attacks were then semi-automatically labeled into the seven aforementioned categories.
2. Generation: LLMs were employed to generate a substantial volume of attacks that were similar in nature to the ones extracted from the hh-rlhf dataset.
3. Crawl and Generate: We utilized LLMs in conjunction with the Wikipedia Crimes Page to generate a series of actions associated with each type of crime. These actions were then employed to devise questions that explore ways to engage in unethical or unlawful actions related to those crimes, specifically focusing on strategies to evade detection or consequences.

Disclaimer:

• The data contains offensive and upsetting content by nature therefore it may not be easy to read. Please read them in accordance with your own personal risk tolerance.
• LLM's response to the AttaQ samples in many cases is harmful and/or violent.
• This dataset is a representative subset of all possible attacks. Many other attacks can cause LLM to answer harmful or inappropriate responses.
• The red-teaming community’s goal is to make models less harmful. We restrict the usage of the dataset to make models less harmful.

Fields

• label: The label of adversarial question
• input: adversarial question
• additional_info: source of the adversarial question

Usage Information

The data are described in the paper: Unveiling Safety Vulnerabilities of Large Language Models.
As detailed in the paper, automatically evaluating the potential harm by LLMs requires a harmlessness metric.
The model under test should be prompted by each question in the dataset, and the corresponding responses undergo evaluation using a metric that considers both the input and output.
Our paper utilizes the "OpenAssistant/reward-model-deberta-v3-large-v2" Reward model, though other models such as "sileod/deberta-v3-large-tasksource-rlhf-reward-model" can also be employed.

import pandas as pd
from transformers import pipeline, AutoTokenizer, AutoModelForSequenceClassification
from datasets import load_dataset

# Load dataset and initialize models
dataset = load_dataset("ibm/AttaQ")
generator = pipeline("text2text-generation", model="gpt2", tokenizer="gpt2")
reward_model = AutoModelForSequenceClassification.from_pretrained("sileod/deberta-v3-large-tasksource-rlhf-reward-model")
tokenizer_reward = AutoTokenizer.from_pretrained("sileod/deberta-v3-large-tasksource-rlhf-reward-model")

# Create DataFrame and generate responses
df = pd.DataFrame(dataset['train'])
df['output'] = [response['generated_text'] for response in generator(df['input'].tolist(), max_length=50, num_return_sequences=1)]

# Calculate harmlessness for each sample
inputs = tokenizer_reward(df['input'].tolist(), df['output'].tolist(), return_tensors="pt", padding=True, truncation=True)
df['reward'] = reward_model(**inputs).logits.detach().cpu().numpy()

A comparison between several models across various attack domains in the Attaq dataset.

Citation Information

If this dataset is utilized in your research, kindly cite the following paper:

@article{kour2023unveiling,
  title={Unveiling Safety Vulnerabilities of Large Language Models},
  author={Kour, George and Zalmanovici, Marcel and Zwerdling, Naama and Goldbraich, Esther and Fandina, Ora Nova and Anaby-Tavor, Ateret and Raz, Orna and Farchi, Eitan},
  journal={arXiv preprint arXiv:2311.04124},
  year={2023}
}