language:

• en
  license: apache-2.0
  task_categories:
• text-generation
  pretty_name: ORPO-DPO-mix-40k-flat
  dataset_info:
  features:
  • name: source
    dtype: string
  • name: chosen
    dtype: string
  • name: rejected
    dtype: string
  • name: prompt
    dtype: string
  • name: system
    dtype: string
  • name: question
    dtype: string
    splits:
  • name: train
    num_bytes: 155571619
    num_examples: 44245
    download_size: 86089921
    dataset_size: 155571619
    configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*
    tags:
• dpo
• rlhf
• preference
• orpo

ORPO-DPO-mix-40k-flat

This dataset is designed for ORPO or DPO training.
See Uncensor any LLM with Abliteration for more information about how to use it.
This is version with raw text instead of lists of dicts as in the original version here.
It makes easier to parse in Axolotl, especially for DPO.

ORPO-DPO-mix-40k-flat is a combination of the following high-quality DPO datasets:

• argilla/Capybara-Preferences: highly scored chosen answers >=5 (7,424 samples)
• argilla/distilabel-intel-orca-dpo-pairs: highly scored chosen answers >=9, not in GSM8K (2,299 samples)
• argilla/ultrafeedback-binarized-preferences-cleaned: highly scored chosen answers >=5 (22,799 samples)
• argilla/distilabel-math-preference-dpo: highly scored chosen answers >=9 (2,181 samples)
• unalignment/toxic-dpo-v0.2 (541 samples)
• M4-ai/prm_dpo_pairs_cleaned (7,958 samples)
• jondurbin/truthy-dpo-v0.1 (1,016 samples)

Rule-based filtering was applied to remove gptisms in the chosen answers (2,206 samples).

Thanks to argilla, unalignment, M4-ai, and jondurbin for providing the source datasets.

🔎 Usage

Here's an example on how to use it as a DPO dataset in Axolotl with ChatML:

rl: dpo
chat_template: chatml
datasets:
  - path: mlabonne/orpo-dpo-mix-40k
    type: chatml.intel

For ORPO, I recommend using mlabonne/orpo-dpo-mix-40k instead.

Toxicity

Note that ORPO-DPO-mix-40k-flat contains a dataset (toxic-dpo-v0.2) designed to prompt the model to answer illegal questions. You can remove it as follows:

dataset = load_dataset('mlabonne/orpo-mix-40k-flat', split='train')
dataset = dataset.filter(
    lambda r: r["source"] != "toxic-dpo-v0.2"
)