dataset_info:
features:
- name: question_id
dtype: string - name: ifeval_prompt
dtype: string - name: key
dtype: int64 - name: instruction_id_list
sequence: string - name: kwargs
list:- name: capital_frequency
dtype: float64 - name: capital_relation
dtype: string - name: end_phrase
dtype: string - name: first_word
dtype: string - name: forbidden_words
sequence: string - name: frequency
dtype: float64 - name: keyword
dtype: string - name: keywords
sequence: string - name: language
dtype: string - name: let_frequency
dtype: float64 - name: let_relation
dtype: string - name: letter
dtype: string - name: nth_paragraph
dtype: float64 - name: num_bullets
dtype: float64 - name: num_highlights
dtype: float64 - name: num_paragraphs
dtype: float64 - name: num_placeholders
dtype: float64 - name: num_sections
dtype: float64 - name: num_sentences
dtype: float64 - name: num_words
dtype: float64 - name: postscript_marker
dtype: string - name: prompt_to_repeat
dtype: string - name: relation
dtype: string - name: section_spliter
dtype: string
- name: capital_frequency
- name: model_name
dtype: string - name: score_data
list:- name: inst_level_loose_acc
sequence: bool - name: inst_level_strict_acc
sequence: bool - name: prompt_level_loose_acc
dtype: bool - name: prompt_level_strict_acc
dtype: bool
- name: inst_level_loose_acc
- name: scores
sequence: bool - name: mean_score
dtype: float64 - name: prompt
dtype: string - name: response_1
dtype: string - name: response_2
dtype: string - name: response_3
dtype: string - name: response_4
dtype: string - name: response_5
dtype: string - name: response_6
dtype: string - name: response_7
dtype: string - name: response_8
dtype: string - name: response_9
dtype: string - name: response_10
dtype: string - name: response_11
dtype: string - name: response_12
dtype: string - name: response_13
dtype: string - name: response_14
dtype: string - name: response_15
dtype: string - name: response_16
dtype: string - name: response_17
dtype: string - name: response_18
dtype: string - name: response_19
dtype: string - name: response_20
dtype: string - name: response_21
dtype: string - name: response_22
dtype: string - name: response_23
dtype: string - name: response_24
dtype: string - name: response_25
dtype: string - name: response_26
dtype: string - name: response_27
dtype: string - name: response_28
dtype: string - name: response_29
dtype: string - name: response_30
dtype: string - name: response_31
dtype: string - name: response_32
dtype: string - name: conflict_pairs
sequence:
sequence: int64 - name: sampled_conflict_pairs
sequence:
sequence: int64
splits: - name: train
num_bytes: 24613605
num_examples: 512
download_size: 13997516
dataset_size: 24613605
configs: - config_name: default
data_files:- split: train
path: data/train-*
- split: train
Overview
This contains the IFEval correctness preference evaluation set for Preference Proxy Evaluations.
The prompts are sampled from IFEval.
This dataset is meant for benchmarking and evaluation, not for training.
License
User prompts are licensed under Apache-2.0, and model outputs are governed by the terms of use set by the respective model providers.
Citation
@misc{frick2024evaluaterewardmodelsrlhf,
title={How to Evaluate Reward Models for RLHF},
author={Evan Frick and Tianle Li and Connor Chen and Wei-Lin Chiang and Anastasios N. Angelopoulos and Jiantao Jiao and Banghua Zhu and Joseph E. Gonzalez and Ion Stoica},
year={2024},
eprint={2410.14872},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2410.14872},
}