dataset_info:
features:

• name: prompt
  dtype: string
• name: prompt_id
  dtype: string
• name: chosen
  list:
  • name: content
    dtype: string
  • name: role
    dtype: string
• name: rejected
  list:
  • name: content
    dtype: string
  • name: role
    dtype: string
• name: messages
  list:
  • name: content
    dtype: string
  • name: role
    dtype: string
• name: score_chosen
  dtype: float64
• name: score_rejected
  dtype: float64
• name: other_info
  struct:
  • name: source
    dtype: string
    splits:
• name: train
  num_bytes: 501881444
  num_examples: 160800
• name: test
  num_bytes: 26966851
  num_examples: 8552
  download_size: 295013949
  dataset_size: 528848295
  configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*
  • split: test
    path: data/test-*

Dataset Card for DPO-hh-rlhf

Reformatted from Anthropic/hh-rlhf dataset.

The LION-series are trained using an empirically optimized pipeline that consists of three stages: SFT, DPO, and online preference learning (online DPO). We find simple techniques such as sequence packing, loss masking in SFT, increasing the preference dataset size in DPO, and online DPO training can significantly improve the performance of language models. Our best models (the LION-series) exceed the performance of the official instruct models tuned with closed-source data and algorithms. To ensure reproducibility, all of our models are trained using publicly available datasets.

For training datasets, code, and evaluation scripts, please refer to our paper and codebase.

Dataset Description

We reformat the Anthropic/hh-rlhf dataset into a common format used across all DPO datasets in this collection. Specifically, we:

1. since the original dataset does not have score ratings, we used "10" for ALL chosen response, and "1" for ALL rejected response as placeholders.