language:
- en
license: apache-2.0
task_categories: - text-generation
pretty_name: ORPO-DPO-mix-40k
dataset_info:
features:- name: source
dtype: string - name: chosen
list:- name: content
dtype: string - name: role
dtype: string
- name: content
- name: rejected
list:- name: content
dtype: string - name: role
dtype: string
- name: content
- name: prompt
dtype: string - name: question
dtype: string
splits: - name: train
num_bytes: 238639013
num_examples: 44245
download_size: 126503374
dataset_size: 238639013
configs:
- name: source
- config_name: default
data_files:- split: train
path: data/train-*
tags:
- split: train
- dpo
- rlhf
- preference
- orpo
ORPO-DPO-mix-40k v1.2
This dataset is designed for ORPO or DPO training.
See Fine-tune Llama 3 with ORPO for more information about how to use it.
It is a combination of the following high-quality DPO datasets:
argilla/Capybara-Preferences: highly scored chosen answers >=5 (7,424 samples)argilla/distilabel-intel-orca-dpo-pairs: highly scored chosen answers >=9, not in GSM8K (2,299 samples)argilla/ultrafeedback-binarized-preferences-cleaned: highly scored chosen answers >=5 (22,799 samples)argilla/distilabel-math-preference-dpo: highly scored chosen answers >=9 (2,181 samples)unalignment/toxic-dpo-v0.2(541 samples)M4-ai/prm_dpo_pairs_cleaned(7,958 samples)jondurbin/truthy-dpo-v0.1(1,016 samples)
Rule-based filtering was applied to remove gptisms in the chosen answers (2,206 samples).
Thanks to argilla, unalignment, M4-ai, and jondurbin for providing the source datasets.
🔎 Usage
v1.2 adds a question column to ensure compatibility with both DPO and ORPO formats in Axolotl.
Here's an example as an ORPO dataset in Axolotl:
rl: orpo
orpo_alpha: 0.1
chat_template: chatml
datasets:
- path: mlabonne/orpo-dpo-mix-40k
type: chat_template.argilla
chat_template: chatml
For DPO, I recommend using mlabonne/orpo-dpo-mix-40k-flat instead.
Toxicity
Note that ORPO-DPO-mix-40k contains a dataset (toxic-dpo-v0.2) designed to prompt the model to answer illegal questions. You can remove it as follows:
dataset = load_dataset('mlabonne/orpo-dpo-mix-40k', split='train')
dataset = dataset.filter(
lambda r: r["source"] != "toxic-dpo-v0.2"
)
History
I'm saving previous versions of this dataset in different branches.