dataset_info:
features:

• name: id
  dtype: int64
• name: key
  dtype: int64
• name: prompt
  dtype: string
• name: instruction_id_list
  sequence: string
• name: kwargs
  list:
  • name: num_highlights
    dtype: int64
  • name: relation
    dtype: string
  • name: num_words
    dtype: int64
  • name: num_placeholders
    dtype: int64
  • name: prompt_to_repeat
    dtype: string
  • name: num_bullets
    dtype: int64
  • name: section_spliter
    dtype: string
  • name: num_sections
    dtype: int64
  • name: capital_relation
    dtype: string
  • name: capital_frequency
    dtype: int64
  • name: keywords
    sequence: string
  • name: num_paragraphs
    dtype: int64
  • name: language
    dtype: string
  • name: let_relation
    dtype: string
  • name: letter
    dtype: string
  • name: let_frequency
    dtype: int64
  • name: end_phrase
    dtype: string
  • name: forbidden_words
    sequence: string
  • name: keyword
    dtype: string
  • name: frequency
    dtype: int64
  • name: num_sentences
    dtype: int64
  • name: postscript_marker
    dtype: string
  • name: first_word
    dtype: string
  • name: nth_paragraph
    dtype: int64
    splits:
• name: eval
  num_bytes: 89240
  num_examples: 148
  download_size: 40293
  dataset_size: 89240
  configs:
• config_name: default
  data_files:
  • split: eval
    path: data/eval.parquet
    license: apache-2.0
    task_categories:
• text-generation
  language:
• en
  pretty_name: IFEval-Hard
  size_categories:
• n<1K

IFEval-Hard

IFEval-Hard is a curated, more challenging, subset of the original IFEval instruction-following benchmark. It is intended for evaluation and stress-testing of instruction-following (and methods that aim to improve it), by focusing on prompts where LLMs (GPT-4o in particular) perform much worse than on the full IFEval dataset.

Paper

Blog Post:

What this is

IFEval is an instruction-following benchmark of 541 samples, where each sample is a prompt containing one or more instructions. Outputs are graded using programmatic checks for whether the instructions were followed.

IFEval-Hard is a subset of 148 challenging examples selected from IFEval to concentrate on cases where the prompt doesn't always elicit a passing response from the LLM.

All models we tested show substantial drops on IFEval-Hard:

How the subset was curated

We define an example as “hard” if GPT-4o fails at least once across 20 runs on that example. We filter the full IFEval set to the examples meeting this criterion, producing a 148-example subset.

This curation targets prompts where frontier and near-frontier models are less saturated, making the subset useful for measuring improvement methods on challenging instruction-following cases.

Intended use

• Evaluating instruction-following robustness on challenging prompts
• Comparing prompt-optimization methods to see which are most effective for getting models to achieve higher pass rates across these more challenging examples, taking out the examples that pass without optimization

What’s included

• A subset of the original IFEval examples (no edits to the underlying example content)
• The dataset preserves the original example structure/fields from IFEval

Relationship to the original dataset

• Parent dataset: google/IFEval
• This dataset: a filtered subset (148 examples) using GPT-4o performance as described above
• No additional annotations are added; evaluation remains programmatic as in IFEval

Dataset Structure

IFEval-Hard retains the same schema as the original google/IFEval dataset with one addition: an id column (0-147) that provides a sequential identifier for this 148-example subset. The original key column from Google's dataset is preserved as-is.

Data Instances

An example from the eval split:

{
    "id": 0,
    "key": 102,
    "prompt": "Write a dialogue between two people, one is dressed up in a ball gown and the other is dressed down in sweats. The two are going to a nightly event. Your answer must contain exactly 3 bullet points in the markdown format (use \"* \" to indicate each bullet) such as:\n* This is the first point.\n* This is the second point.",
    "instruction_id_list": [
        "detectable_format:number_bullet_lists"
    ],
    "kwargs": [
        {
            "num_highlights": null,
            "relation": null,
            "num_words": null,
            "num_placeholders": null,
            "prompt_to_repeat": null,
            "num_bullets": 3,
            "section_spliter": null,
            "num_sections": null,
            "capital_relation": null,
            "capital_frequency": null,
            "keywords": null,
            "num_paragraphs": null,
            "language": null,
            "let_relation": null,
            "letter": null,
            "let_frequency": null,
            "end_phrase": null,
            "forbidden_words": null,
            "keyword": null,
            "frequency": null,
            "num_sentences": null,
            "postscript_marker": null,
            "first_word": null,
            "nth_paragraph": null
        }
    ]
}

Data Fields

• id: Sequential identifier for the IFEval-Hard subset (0-147). This is the only column added relative to the original google/IFEval dataset.
• key: The unique ID for the prompt from the original google/IFEval dataset, preserved to maintain traceability back to the source.
• prompt: Describes the task the model should perform.
• instruction_id_list: An array of verifiable instructions. See Table 1 of the IFEval paper for the full set with their descriptions.
• kwargs: An array of arguments used to specify each verifiable instruction in instruction_id_list.

Citation

@misc{singhal2026prefpopairwisepreferenceprompt,
      title={PrefPO: Pairwise Preference Prompt Optimization}, 
      author={Rahul Singhal and Pradyumna Tambwekar and Karime Maamari},
      year={2026},
      eprint={2603.19311},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.19311}, 
}