dataset_info:
features:

• name: id
  dtype: string
• name: prompt
  dtype: string
• name: chosen
  list:
  • name: content
    dtype: string
  • name: role
    dtype: string
• name: rejected
  list:
  • name: content
    dtype: string
  • name: role
    dtype: string
    splits:
• name: train
  num_bytes: 431879752
  num_examples: 65551
  download_size: 225722195
  dataset_size: 431879752
  configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*

Llama 3.1 Tulu 3 IF-Augmented (on-policy 70B)

Note that this collection is licensed under ODC-BY-1.0 license; different licenses apply to subsets of the data. Some portions of the dataset are non-commercial. We present the mixture as a research artifact.

This preference dataset is part of our Tulu 3 preference mixture:
it contains prompts from SFT Data, with constraints from https://huggingface.co/datasets/google/IFEval. It contains 65,530 generation pairs (some of which on-policy completions from https://huggingface.co/allenai/Llama-3.1-Tulu-3-70B) obtained using the following models:

• Mistral 7B Instruct v0.2 (Apache 2.0)
• Mistral Nemo Instruct 2407 (Apache 2.0)
• Tulu 2 7B (Ai2 ImpACT Low Risk License)
• Tulu 2 13B (Ai2 ImpACT Low Risk License)
• Yi-34B-Chat (Apache 2.0)
• Yi-6B-Chat (Apache 2.0)
• MPT 30B Chat (CC-BY-SA-4.0)
• MPT 7B 8k Chat (CC-BY-SA-4.0)
• Google Gemma 2 27B it (Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms)
• Google Gemma 2 9B it (Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms)
• InternLM2.5 20B (InternLM weights are fully open for academic research and also allow free commercial usage. A commercial license can be obtained as instructed in the model card.)
• InternLM2.5 7B (InternLM weights are fully open for academic research and also allow free commercial usage. A commercial license can be obtained as instructed in the model card.)
• InternLM2.5 1.8B (InternLM weights are fully open for academic research and also allow free commercial usage. A commercial license can be obtained as instructed in the model card.)
• Falcon 7B (Apache 2.0)
• Qwen2.5 72B Instruct (Qwen is licensed under the Qwen LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved.)
• Qwen2.5 32B Instruct (Apache 2.0)
• Qwen2.5 14B Instruct (Apache 2.0)
• Qwen2.5 7B Instruct (Apache 2.0)
• Llama 3.1 8B Instruct (this dataset was partially "Built with Llama" and is thus subject to the Llama 3.1 License)
• Llama 3.1 70B Instruct (this dataset was partially "Built with Llama" and is thus subject to the Llama 3.1 License)
• Llama 3 8B Instruct (this dataset was partially "Built with Meta Llama 3" and is thus subject to the Llama 3 License)
• GPT-4 Turbo and GPT-4o (Outputs produced by GPT-4 are subject to OpenAI's terms of use)
• Claude 3.5 Sonnet (Outputs produced by Claude are subject to Anthropic terms of service and usage policy)

Completion Generation Approach:

Given a set of prompts, we generated the completions and preferences using a synthetic pipeline that combines both on-policy and off-policy data, and obtained the preference annotations on four different aspects using the Ultrafeedback template and an LLM judge. The code for the synthetic generation pipeline is found in the scripts/synth_pref directory of open-instruct

License

This dataset is licensed under ODC-BY. It is intended for research and educational use in accordance with Ai2's Responsible Use Guidelines. This dataset includes output data generated from third party models that are subject to separate terms governing their use.