dataset_info:
features:

• name: key
  dtype: int64
• name: prompt
  dtype: string
• name: instruction_id_list
  list: string
• name: kwargs
  list:
  • name: num_highlights
    dtype: int64
  • name: relation
    dtype: string
  • name: num_words
    dtype: int64
  • name: num_placeholders
    dtype: int64
  • name: prompt_to_repeat
    dtype: string
  • name: num_bullets
    dtype: int64
  • name: section_spliter
    dtype: string
  • name: num_sections
    dtype: int64
  • name: capital_relation
    dtype: string
  • name: capital_frequency
    dtype: int64
  • name: keywords
    list: string
  • name: num_paragraphs
    dtype: int64
  • name: language
    dtype: string
  • name: let_relation
    dtype: string
  • name: letter
    dtype: string
  • name: let_frequency
    dtype: int64
  • name: end_phrase
    dtype: string
  • name: forbidden_words
    list: string
  • name: keyword
    dtype: string
  • name: frequency
    dtype: int64
  • name: num_sentences
    dtype: int64
  • name: postscript_marker
    dtype: string
  • name: first_word
    dtype: string
  • name: nth_paragraph
    dtype: int64
    splits:
• name: test
  num_bytes: 344197
  num_examples: 541
  download_size: 99799
  dataset_size: 344197
  configs:
• config_name: default
  data_files:
  • split: test
    path: data/test-*
    license: apache-2.0
    language:
• en
  size_categories:
• n<1K

IFEval Dataset Card

Dataset Description

IFEval is an instruction-following evaluation benchmark consisting of verifiable natural language instructions. Each example specifies one or more constraints that a model must satisfy in its output (e.g., include/exclude phrases, follow a format, respect length or style constraints). In this project, IFEval is used to evaluate both:

• instruction following in the reasoning trace (RT), and
• instruction following in the final answer (FA).

The benchmark provides automatic, rule-based checks so that compliance can be measured without human annotation.

Intended Use

• Primary use: Evaluate how well language and reasoning models follow explicit instructions under objective, automatically checkable criteria.
• This repository’s use: Measure instruction-following performance of controllable reasoning models in both their RTs and FAs.

The dataset is intended only for research and benchmarking and not for deployment in user-facing applications as-is.

Dataset Structure

• Total size: 541 instructions
• Splits: only test

Each instance conceptually contains:

• key: id of the data point
• prompt The user prompt. We have added an instruction at the end stating that the instructions applies to both the reasoning traces and final answers.
• instruction_id_list / metadata: machine-readable specification of constraints used by the evaluation scripts.
• kwargs: information needed by the evaluation script to verify whether constraints are met.

Tasks and Evaluation

• Main task: Instruction following.
• Evaluation metric: Instruction-level loose-accuracy, as defined in the original IFEval publication. A prediction is considered correct if it satisfies the verifiable constraints associated with the instruction.
• In this work:
  • RTs and FAs are evaluated separately with the same loose-accuracy metric to obtain IF-RT and IF-FA scores.

Data Source

IFEval was introduced as an automatic benchmark for instruction following in LLMs. For details on construction, examples, and official evaluation scripts, please refer to the original IFEval paper and repository.

License

• License: Apache 2.0

Known Limitations and Considerations

• The benchmark focuses on objective, verifiable instructions and may not capture more nuanced forms of instruction following (e.g., subtle style preferences, conversational alignment).
• All instructions are in English, so the dataset does not directly measure multilingual instruction-following capabilities.
• Since the test set is relatively small (540 examples), fine-grained comparisons between very similar systems should be interpreted with care.

Citation

If you use this dataset in your work, please cite the original IFEval paper and this repository’s paper:

@misc{puerto2026leakythoughtsprivatereasoning,
      title={From Leaky Thoughts to Private Reasoning: Controlling What LRMs Say to Themselves}, 
      author={Haritz Puerto and Haonan Li and Xudong Han and Timothy Baldwin and Iryna Gurevych},
      year={2026},
      eprint={2602.24210},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.24210}, 
}

@misc{zhou2023instructionfollowingevaluationlargelanguage,
      title={Instruction-Following Evaluation for Large Language Models}, 
      author={Jeffrey Zhou and Tianjian Lu and Swaroop Mishra and Siddhartha Brahma and Sujoy Basu and Yi Luan and Denny Zhou and Le Hou},
      year={2023},
      eprint={2311.07911},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2311.07911}, 
}