configs:
- config_name: memwrap
data_files:- split: test
path: memwrap/test-*
- split: test
- config_name: plain
data_files:- split: test
path: plain/test-*
dataset_info:
- split: test
- config_name: memwrap
features:- name: prompt
dtype: string - name: category
dtype: string - name: extra_info
struct:- name: ground_truth
struct:- name: instruction_id_list
list: string - name: kwargs
list:- name: num_highlights
dtype: int64 - name: relation
dtype: string - name: num_words
dtype: int64 - name: num_placeholders
dtype: int64 - name: prompt_to_repeat
dtype: string - name: num_bullets
dtype: int64 - name: section_spliter
dtype: string - name: num_sections
dtype: int64 - name: capital_relation
dtype: string - name: capital_frequency
dtype: int64 - name: keywords
list: string - name: num_paragraphs
dtype: int64 - name: language
dtype: string - name: let_relation
dtype: string - name: letter
dtype: string - name: let_frequency
dtype: int64 - name: end_phrase
dtype: string - name: forbidden_words
list: string - name: keyword
dtype: string - name: frequency
dtype: int64 - name: num_sentences
dtype: int64 - name: postscript_marker
dtype: string - name: first_word
dtype: string - name: nth_paragraph
dtype: int64
- name: num_highlights
- name: key
dtype: int64
- name: instruction_id_list
- name: scoring_function
dtype: string - name: variant
dtype: string
splits:
- name: ground_truth
- name: test
num_bytes: 440490
num_examples: 541
download_size: 109267
dataset_size: 440490
- name: prompt
- config_name: plain
features:- name: prompt
dtype: string - name: category
dtype: string - name: extra_info
struct:- name: ground_truth
struct:- name: instruction_id_list
list: string - name: key
dtype: int64 - name: kwargs
list:- name: capital_frequency
dtype: int64 - name: capital_relation
dtype: string - name: end_phrase
dtype: string - name: first_word
dtype: string - name: forbidden_words
list: string - name: frequency
dtype: int64 - name: keyword
dtype: string - name: keywords
list: string - name: language
dtype: string - name: let_frequency
dtype: int64 - name: let_relation
dtype: string - name: letter
dtype: string - name: nth_paragraph
dtype: int64 - name: num_bullets
dtype: int64 - name: num_highlights
dtype: int64 - name: num_paragraphs
dtype: int64 - name: num_placeholders
dtype: int64 - name: num_sections
dtype: int64 - name: num_sentences
dtype: int64 - name: num_words
dtype: int64 - name: postscript_marker
dtype: string - name: prompt_to_repeat
dtype: string - name: relation
dtype: string - name: section_spliter
dtype: string
- name: capital_frequency
- name: instruction_id_list
- name: scoring_function
dtype: string - name: variant
dtype: string
splits:
- name: ground_truth
- name: test
num_bytes: 387472
num_examples: 541
download_size: 104139
dataset_size: 387472
- name: prompt
IFEval: Instruction-Following Evaluation
Benchmark for testing LLMs' ability to follow verifiable instructions with precise constraints.
Overview
| Metric | Value |
|---|---|
| Prompts | 541 |
| Constraint Types | 25+ |
| Constraints per Prompt | 1-3 |
| Evaluation Method | Automatic verification |
Constraint Categories
IFEval tests 25+ types of verifiable constraints across multiple categories:
Format Constraints
- JSON format: Response must be valid JSON
- Bullet lists: Required number of bullet-point lists
- Highlighted sections: Markdown highlighted text sections
- Title: Response must include a title
- Multiple sections: Required number of sections/headers
- Constrained response: Response limited to specific options
Length Constraints
- Word count: Exactly, at least, or at most N words
- Sentence count: Required number of sentences
- Paragraph count: Required number of paragraphs
- Nth paragraph first word: Specific first word for paragraph N
Keyword Constraints
- Keyword existence: Required keywords must appear
- Keyword frequency: Keyword appears N times
- Forbidden words: Certain words must not appear
- Letter frequency: Specific letter appears N times
Style Constraints
- All uppercase: Entire response in capital letters
- All lowercase: Entire response in lowercase
- Capital word frequency: Required number of capitalized words
- No comma: Response must not contain commas
Content Constraints
- Placeholders: Required number of [placeholder] elements
- Postscript: Must include P.S. or P.P.S.
- Quotation: Entire response wrapped in quotes
- Two responses: Must provide two separate answers
- Repeat prompt: Must repeat the original prompt
Position Constraints
- End phrase: Response must end with specific phrase
- Language: Response in specified language
Source
Based on IFEval benchmark by Google Research.
Paper: Instruction-Following Evaluation for Large Language Models
Variants
- memwrap: Instructions wrapped with
<|memory_start|>/<|memory_end|>tags + meta-instruction - plain: Raw instruction prompts without memory tags
Usage
from datasets import load_dataset
# Load memwrap variant
ds = load_dataset("tonychenxyz/ifeval", "memwrap", split="test")
# Load plain variant
ds = load_dataset("tonychenxyz/ifeval", "plain", split="test")
# Example entry
print(ds[0])
# {
# "prompt": "<formatted prompt with chat template>",
# "category": "memwrap/ifeval",
# "extra_info": {
# "ground_truth": {
# "instruction_id_list": ["punctuation:no_comma", "length_constraints:number_words"],
# "kwargs": [{}, {"relation": "at least", "num_words": 100}],
# "key": 0
# },
# "scoring_function": "ifeval",
# "variant": "memwrap"
# }
# }
Scoring
Uses ifeval scoring function with automatic constraint verification:
Metrics:
strict_accuracy: All constraints satisfied (1.0 or 0.0)loose_accuracy: At least one constraint satisfied (1.0 or 0.0)num_constraints: Total number of constraintsnum_passed: Number of constraints satisfiedconstraint_N_pass: Individual constraint results (1.0 or 0.0)
Example:
{
"strict_accuracy": 0.0,
"loose_accuracy": 1.0,
"num_constraints": 2.0,
"num_passed": 1.0,
"constraint_1_pass": 0.0,
"constraint_2_pass": 1.0
}
Citation
@article{zhou2023instruction,
title={Instruction-Following Evaluation for Large Language Models},
author={Zhou, Jeffrey and Lu, Tianjian and Mishra, Swaroop and Brahma, Siddhartha and Basu, Sujoy and Luan, Yi and Zhou, Denny and Hou, Le},
journal={arXiv preprint arXiv:2311.07911},
year={2023}
}