tags:

• Instruction_Following
  size_categories:
• 1K<n<10K

Overview

Inverse IFEval is a novel benchmark designed to evaluate large language models' (LLMs) ability to follow counterintuitive instructions that deliberately deviate from conventional training paradigms. The dataset challenges models to override their ingrained training conventions and faithfully execute instructions that conflict with standard cognitive patterns or annotation norms.

Key Features

• Counter-Cognitive Evaluation: Measures models' ability to suppress training-induced biases and follow unconventional instructions
• Eight Instruction Types: Systematically designed categories that invert standard training paradigms
• Multilingual Support: Balanced Chinese and English versions (506 samples each)
• Diverse Domains: Covers 23 knowledge domains including computer science, mathematics, law, and biology
• High-Quality Construction: Multi-stage human-in-the-loop pipeline with expert verification

Dataset Composition

The dataset contains 1,012 high-quality questions distributed across eight instruction types:

Construction Methodology

The dataset was built through a rigorous five-stage process:

1. Observation & Reversal: Analyze and invert standard SFT paradigms
2. Seed Data Construction: Expert-crafted seed questions
3. Large-Scale Generation: LLM-assisted question generation
4. Automatic Filtering: Quality control mechanisms
5. Human Verification: Expert review and calibration

Evaluation Protocol

• Uses "LLM-as-a-Judge" paradigm with 98% accuracy
• Adaptive judge model matrix for different instruction types
• Optimized judging templates and system prompts
• Scores range 0-100 based on instruction-following fidelity

"LLM-as-a-Judge" pipeline

Our automated evaluation framework, as illustrated in the figure, follows a structured "LLM-as-a-Judge" approach. The pipeline consists of the following steps:

1. Generation: An input prompt from the evaluation dataset is provided to the Model Under Evaluation (the target model being tested). The model generates a response based on this prompt.
2. Input Assembly for Judge: The generated response is then prepared for assessment. We construct a comprehensive input for the Judge Model by populating a Judge Prompt Template. This template integrates the model's response with a ground-truth Reference Answer and Scoring Criteria that outline the key points for a correct answer.
3. Evaluation: The assembled prompt, along with a high-level Judge System Prompt (which instructs the judge on its task, persona, and output format), is fed into the Judge Model.
4. Output: The Judge Model processes the inputs and delivers its verdict, which includes a numerical score and a textual Evaluation Rationale justifying the given score.

Judge Model Matrix

Here is the optimal judge model and judging template structure corresponding to various types of instructions:

Key Findings

• Top-performing model (o3-high) achieves 75.66 overall score
• Models show significant performance drops (∼30%) on counterintuitive vs conventional instructions
• Thinking mechanisms improve performance by ∼15% on average
• Current alignment methods struggle with cognitive inertia

Intended Use

• Evaluating instruction-following robustness
• Testing model flexibility beyond training distributions
• Identifying limitations in current alignment methods
• Developing more adaptable LLMs

Citation

When using this dataset, please cite the original work:

@misc{zhang2025inverseifevalllmsunlearn,
      title={Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?}, 
      author={Qinyan Zhang and Xinping Lei and Ruijie Miao and Yu Fu and Haojie Fan and Le Chang and Jiafan Hou and Dingling Zhang and Zhongfei Hou and Ziqiang Yang and Changxin Pu and Fei Hu and Jingkai Liu and Mengyun Liu and Yang Liu and Xiang Gao and Jiaheng Liu and Tong Yang and Zaiyuan Wang and Ge Zhang and Wenhao Huang},
      year={2025},
      eprint={2509.04292},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.04292}, 
}