hendrydong/preference_700K

dataset_info:
features:

• name: rejected
  list:
  • name: content
    dtype: string
  • name: role
    dtype: string
• name: rejected_score
  dtype: float64
• name: chosen_score
  dtype: float64
• name: chosen
  list:
  • name: content
    dtype: string
  • name: role
    dtype: string
    splits:
• name: train
  num_bytes: 2802733004
  num_examples: 700000
  download_size: 1433169364
  dataset_size: 2802733004
  configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*

This is the preference dataset used for RLHF Workflow project.

It is a mixed dataset of the following complenent,

• HH-RLHF (Bai et al., 2022a) is a pairwise preference dataset where each sample is accompanied
  by a conversation history and two alternative responses written by an early Claude model with 52B
  parameters. The preferences of the responses are annotated by humans.
• SHP (Ethayarajh et al., 2022) is sourced from Reddit and includes examples from 18 subreddits, such
  as askacademia, askbaking, askengineers, and changemyview. Each example is a Reddit post with a
  question/instruction and a pair of top-level comments. One comment is preferred by more Reddit users
  than the other. All preferences and responses are provided by humans. Only samples with a score
  ratio > 2 are used, and at most 5 pairs are taken for each prompt.
• HelpSteer (Wang et al., 2023). This open-source dataset (Wang et al., 2023) contains prompts,
  responses, and five human-annotated attributes (helpfulness, correctness, coherence, complexity, and
  verbosity) ranging from 0 to 4. The prompts are generated using a mixture of template-generated and
  human-generated methods, while responses are generated by an in-house LLM. The authors generate
  up to 4 responses per prompt, and we can construct pairwise comparisons based on them.
• PKU-SafeRLHF (Ji et al., 2024). This dataset (Ji et al., 2024) consists of 30k+ expert comparison
  data. Each sample includes two responses to a question and two preference signals for helpfulness and
  safety, respectively. The responses are generated by open-source chatbots, and the preference signals
  are merged through the results of 14 harm category multi-class classficiation.
• UltraFeedback (Cui et al., 2023) consists of 64k prompts from diverse resources (including UltraChat,
  ShareGPT, Evol-Instruct, TruthfulQA, FalseQA, and FLAN) and the authors generate 4 responses per
  prompt using 4 different LLMs sampled from a diverse set of state-of-the-art open-source LLMs. The
  preference is from GPT-4 based on a fine-grained annotation instruction, which contains 4 different
  aspects, namely instruction-following, truthfulness, honesty and helpfulness. The dataset collection
  strategy of UltraFeedback has also influenced many subsequent works.
• UltraInteract (Yuan et al., 2024a) is a preference dataset designed for complex reasoning tasks. The
  authors collect a preference tree for each instruction, with the instruction being the root and each
  action a node. A trajectory is a root-to-leaf path consisting of a sequence of actions. Paired correct
  and incorrect nodes or trajectories are used for preference learning.
• Distilabel-Capybara is a preference dataset of multi-turn dialogues whose prompts are taken from
  Daniele and Suphavadeeprasit (2023), where the responses are generated by open-source LLMs and
  preferences are generated by GPT-4.
• Distilabel-Orca is collected similarly with Capybara but with the prompts from Lian et al. (2023a).

Reference

If you found it useful, please cite

@misc{dong2024rlhf,
      title={RLHF Workflow: From Reward Modeling to Online RLHF}, 
      author={Hanze Dong and Wei Xiong and Bo Pang and Haoxiang Wang and Han Zhao and Yingbo Zhou and Nan Jiang and Doyen Sahoo and Caiming Xiong and Tong Zhang},
      year={2024},
      eprint={2405.07863},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}