license: apache-2.0
task_categories:
- conversational
language: - en
tags: - harmless
size_categories: - 10K<n<100K
Dataset Card for Anthropic_HH_Golden
This dataset is constructed to test the ULMA technique as mentioned in the paper Unified Language Model Alignment with Demonstration and Point-wise Human Preference (under review, and an arxiv link will be provided soon). They show that replacing the positive samples in a preference dataset by high-quality demonstration data (golden data) greatly improves the performance of various alignment methods (RLHF, DPO, ULMA). In particular, the ULMA method exploits the high-quality demonstration data in the preference dataset by treating the positive and negative samples differently, and boosting the performance by removing the KL regularizer for positive samples.
Dataset Summary
This repository contains a new preference dataset extending the harmless dataset of Anthropic's Helpful and Harmless (HH) datasets. The origin positive response in HH is generated by a supervised fined-tuned model of Anthropic, where harmful and unhelpful responses are freqently encountered. In this dataset, the positive responses are replaced by re-rewritten responses generated by GPT4.
Comparison with the origin HH dataset. Left is the data sampled from the origin HH dataset, and right is the corresponding answer in our Anthropic_HH_Golden dataset. The highlighted parts are the differences. It is clear that after the rewritten, the "chosen" responses is more harmless, and the "rejected" response are left unchanged.
Usage
from datasets import load_dataset
# Load the harmless dataset with golden demonstration
dataset = load_dataset("Unified-Language-Model-Alignment/Anthropic_HH_Golden")
or download the data files directly with:
git clone https://huggingface.co/datasets/Unified-Language-Model-Alignment/Anthropic_HH_Golden