dataset_info:
features:
- name: key
dtype: int64 - name: prompt
dtype: string - name: instruction_id_list
list: string - name: kwargs
list:- name: num_highlights
dtype: int64 - name: relation
dtype: string - name: num_words
dtype: int64 - name: num_placeholders
dtype: int64 - name: prompt_to_repeat
dtype: string - name: num_bullets
dtype: int64 - name: section_spliter
dtype: string - name: num_sections
dtype: int64 - name: capital_relation
dtype: string - name: capital_frequency
dtype: int64 - name: keywords
list: string - name: num_paragraphs
dtype: int64 - name: language
dtype: string - name: let_relation
dtype: string - name: letter
dtype: string - name: let_frequency
dtype: int64 - name: end_phrase
dtype: string - name: forbidden_words
list: string - name: keyword
dtype: string - name: frequency
dtype: int64 - name: num_sentences
dtype: int64 - name: postscript_marker
dtype: string - name: first_word
dtype: string - name: nth_paragraph
dtype: int64
splits:
- name: num_highlights
- name: test
num_bytes: 278553
num_examples: 541
download_size: 94981
dataset_size: 278553
configs: - config_name: default
data_files:- split: test
path: data/test-*
license: apache-2.0
task_categories:
- split: test
- text-generation
language: - en
size_categories: - n<1K
English Instruction-Following Eval (IFEval-en)
The original IFEval benchmark (Zhou et al., 2023) contains a number of examples with the constraint requiring the response to be
in a certain language. While generally this doesn't pose a problem for large multilingual models, it penalizes smaller specialized models
that are not expected to be fluent in all these additional languages. Accordingly, IFEval-en introduces the following changes
to google/IFEval to ensure that the benchmark measures instruction following
capabilities specifically in English:
- Removed/replaced constraints requesting specific language of response
- Fixed the example with key 1174 based on jzhang86/corrected_ifeval