configs:
- config_name: default
data_files:- split: en
path:- data/catqa_english.json
- split: zh
path:- data/catqa_chinese.json
- split: vi
path:- data/catqa_vietnamese.json
license: apache-2.0
language:
- data/catqa_vietnamese.json
- split: en
- en
- zh
- vi
size_categories: - 1K<n<10K
license: apache-2.0
CatQA: A categorical harmful questions dataset
- CatQA is used in LLM safety realignment research:
Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic (Paper, Code)
How to download
from datasets import load_dataset
dataset = load_dataset("declare-lab/CategoricalHarmfulQA")
What is CatQA?
To comprehensively evaluate the model across a wide range of harmful categories, we construct a new safety evaluation dataset CatQA. We obtain a combined list of prohibited use cases mentioned in OpenAI usage policies and Meta’s Llama2 acceptable use policy. As shown in the table below, there are 11 main categories of harm we consider. We divide each of these categories into 5 sub-categories. For each sub-category, we frame 10 harmful questions. Each sub-category and question is collected by us with the assistance of a powerful but unaligned LLM (the LLM is helpful but not safety-aligned. We do not disclose the name of LLM to prevent its misuse}. Thus, the dataset consists of 550 harmful questions, 55 such questions are shown in the table. While CatQA questions are in English, to evaluate the generalizability of our approach, we extend the dataset to two more languages: Chinese and Vietnamese. First, we translated the English version to the corresponding language using the LLM which assisted us in generating the former version, and employed language-proficient human annotators to enhance the quality of the dataset.
Citation
@misc{bhardwaj2024language,
title={Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic},
author={Rishabh Bhardwaj and Do Duc Anh and Soujanya Poria},
year={2024},
eprint={2402.11746},
archivePrefix={arXiv},
primaryClass={cs.CL}
}