language:

• en
  license: apache-2.0
  dataset_info:
  features:
  • name: system
    dtype: string
  • name: prompt
    dtype: string
  • name: completion
    dtype: string
  • name: label
    dtype: bool
  • name: rating
    dtype: float64
  • name: status
    dtype: string
  • name: in_gsm8k_train
    dtype: bool
    splits:
  • name: train
    num_bytes: 41940501
    num_examples: 23147
    download_size: 17134184
    dataset_size: 41940501
    configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*
    tags:
• rlaif
• kto
• rlhf
• distilabel
• synthetic

distilabel Orca Pairs for KTO

A KTO signal transformed version of the highly loved distilabel Orca Pairs for DPO.

The dataset is a "distilabeled" version of the widely used dataset: Intel/orca_dpo_pairs. The original dataset has been used by 100s of open-source practitioners and models. We knew from fixing UltraFeedback (and before that, Alpacas and Dollys) that this dataset could be highly improved.

Continuing with our mission to build the best alignment datasets for open-source LLMs and the community, we spent a few hours improving it with distilabel.

This was our main intuition: the original dataset just assumes gpt4/3.5-turbo are always the best response. We know from UltraFeedback that's not always the case. Moreover, DPO fine-tuning benefits from the diversity of preference pairs.

Additionally, we have added a new column indicating whether the question in the dataset is part of the train set of gsm8k (there were no examples from the test set). See the reproduction section for more details.

Why KTO?

The KTO paper states:

• KTO matches or exceeds DPO performance at scales from 1B to 30B parameters.1 That is, taking a preference dataset of n DPO pairs and breaking it up into 2n examples for KTO can yield better generations, despite the model ostensibly learning from a weaker signal.
• KTO can handle extreme data imbalances, matching DPO performance while using up to 90% fewer desirable examples (i.e., examples of good generations). Its success thus cannot be ascribed to the alignment data being sourced from a preference dataset.
• When the pretrained model is sufficiently good, one can skip supervised finetuning and go straight to KTO without a loss in generation quality. In contrast, we find that without doing SFT first, DPO-aligned models are significantly worse at all scales.

Reproduce KTO Transformation

Orginal distilabel intel orca dpo pairs