license: apache-2.0
language:

• en
  size_categories:
• n<1K
  task_categories:
• text-generation
  pretty_name: Split IFEval

Split IFEval

This dataset modifies the Instruction-Following Eval (IFEval) benchmark to split apart the task from the syntactic instructions in addition to fixing errors in the original dataset.
It enables the use of research methods like attention steering that require access to the instruction text.

To load the dataset, run:

from datasets import load_dataset

split_ifeval = load_dataset("ibm-research/Split-IFEval")

Dataset Structure

Each entry in the dataset includes the following fields:

1. key: Unique sample identifier
2. prompt: Prompt where the task and instructions are split
3. instruction_id_list: Verifiable instruction identifiers for evaluation
4. kwargs: Arguments associated with each instruction identifier
5. separated_prompt: The task prompt
6. instructions: The list of instructions
7. original_prompt: The original IFEval prompt

Here is an example:

{
        "key": 1000,
        "prompt": "Write a summary of the wikipedia page \"https://en.wikipedia.org/wiki/Raymond_III,_Count_of_Tripoli\".\n\nYour response should follow the instructions below:\n- Write 300+ words\n- Do not use any commas\n- Highlight at least 3 sections that have titles in markdown format, for example *highlighted section part 1*, *highlighted section part 2*, *highlighted section part 3*",
        "instruction_id_list": [
            "punctuation:no_comma",
            "detectable_format:number_highlighted_sections",
            "length_constraints:number_words"
        ],
        "kwargs": [
            {},
            {
                "num_highlights": 3
            },
            {
                "relation": "at least",
                "num_words": 300
            }
        ],
        "separated_prompt": "Write a summary of the wikipedia page \"https://en.wikipedia.org/wiki/Raymond_III,_Count_of_Tripoli\".",
        "instructions": [
            "- Write 300+ words",
            "- Do not use any commas",
            "- Highlight at least 3 sections that have titles in markdown format, for example *highlighted section part 1*, *highlighted section part 2*, *highlighted section part 3*"
        ],
        "original_prompt": "Write a 300+ word summary of the wikipedia page \"https://en.wikipedia.org/wiki/Raymond_III,_Count_of_Tripoli\". Do not use any commas and highlight at least 3 sections that has titles in markdown format, for example *highlighted section part 1*, *highlighted section part 2*, *highlighted section part 3*."
}

Licensing Information

The dataset is available under the Apache 2.0 license.

Citation

Please consider citing our work if you use this dataset.

@article{venkateswaran2025spotlight,
  title={Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering},
  author={Venkateswaran, Praveen and Contractor, Danish},
  journal={arXiv preprint arXiv:2505.12025},
  year={2025}
}