language:

• en
  license: apache-2.0
  task_categories:
• text-generation
  pretty_name: ORPO-DPO-mix-40k
  dataset_info:
  features:
  • name: source
    dtype: string
  • name: chosen
    list:
    • name: content
      dtype: string
    • name: role
      dtype: string
  • name: rejected
    list:
    • name: content
      dtype: string
    • name: role
      dtype: string
  • name: prompt
    dtype: string
  • name: question
    dtype: string
    splits:
  • name: train
    num_bytes: 238639013
    num_examples: 44245
    download_size: 126503374
    dataset_size: 238639013
    configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*
    tags:
• dpo
• rlhf
• preference
• orpo

ORPO-DPO-mix-40k v1.2

This dataset is designed for ORPO or DPO training.
See Fine-tune Llama 3 with ORPO for more information about how to use it.
It is a combination of the following high-quality DPO datasets:

• argilla/Capybara-Preferences: highly scored chosen answers >=5 (7,424 samples)
• argilla/distilabel-intel-orca-dpo-pairs: highly scored chosen answers >=9, not in GSM8K (2,299 samples)
• argilla/ultrafeedback-binarized-preferences-cleaned: highly scored chosen answers >=5 (22,799 samples)
• argilla/distilabel-math-preference-dpo: highly scored chosen answers >=9 (2,181 samples)
• unalignment/toxic-dpo-v0.2 (541 samples)
• M4-ai/prm_dpo_pairs_cleaned (7,958 samples)
• jondurbin/truthy-dpo-v0.1 (1,016 samples)

Rule-based filtering was applied to remove gptisms in the chosen answers (2,206 samples).

Thanks to argilla, unalignment, M4-ai, and jondurbin for providing the source datasets.

🔎 Usage

v1.2 adds a question column to ensure compatibility with both DPO and ORPO formats in Axolotl.

Here's an example as an ORPO dataset in Axolotl:

rl: orpo
orpo_alpha: 0.1
chat_template: chatml
datasets:
  - path: mlabonne/orpo-dpo-mix-40k
    type: chat_template.argilla
    chat_template: chatml

For DPO, I recommend using mlabonne/orpo-dpo-mix-40k-flat instead.

Toxicity

Note that ORPO-DPO-mix-40k contains a dataset (toxic-dpo-v0.2) designed to prompt the model to answer illegal questions. You can remove it as follows:

dataset = load_dataset('mlabonne/orpo-dpo-mix-40k', split='train')
dataset = dataset.filter(
    lambda r: r["source"] != "toxic-dpo-v0.2"
)

History

I'm saving previous versions of this dataset in different branches.

• v1.0