license: mit
language:
- en
size_categories: - 100K<n<1M
The Hermes_preference dataset
The Hermes_preference dataset is the type of feedback dataset, used for training reward models which is used for RLHF!
In addition, Hermes_preference dataset can be also used for DPO!
We collect the preference data from several popular feedback datasets(UltraFeedback, hh-rlhf, rlhf-reward-datasets) through sampling and preprocessing.
As a result, we could have collected approximately 190K preference data.
To collect high-quality feedback data, we decided to collect feedback data from UltraFeedback & rlhf-reward-datasets which are curated datasets.
In addition, we also collect the data from hh-rlhf to accumulate the data that teach the models to output helpful and harmless response.
We hope that Hermes_preference dataset provides a promising way to future RLHF & DPO research!
Dataset Details
The Hermes_preference dataset is a mixture of several popular preference datasets(UltraFeedback, hh-rlhf, rlhf-reward-datasets) as we mentioned above.
The purpose of this dataset is to make a preference dataset that consists of more varied data.
To accomplish this purpose, we selected the UltraFeedback, hh-rlhf, and rlhf-reward-datasets as the base dataset.
More specifically, we sampled and preprocessed the datasets mentioned above to make Hermes_preference dataset more structural.
- Curated by: [More Information Needed]
- Language(s) (NLP): en
- License: MIT
Source Data
The Hermes_preference dataset consists of the following datasets.
Dataset Sources [optional]
- Repository: gauss5930/Hermes
- Model Cartinoe5930/Hermes-7b
Dataset Structure
The structure of Hermes_prference dataset is as follows:
{
"source": The source dataset of data,
"prompt": The instruction of question,
"chosen": Choosed response,
"rejected": Rejected response
}