license: odc-by
task_categories:

• text-generation
• reinforcement-learning
  language:
• en
  tags:
• evaluation
• ifeval
• instruction-following
• reasoning
• reinforcement-learning
• rlhf
• verl
  size_categories:
• 10K<n<100K
  pretty_name: IFBench-VERL (Instruction Following Evaluation)
  dataset_info:
  features:
  • name: data_source
    dtype: string
  • name: prompt
    list:
    • name: role
      dtype: string
    • name: content
      dtype: string
  • name: ability
    dtype: string
  • name: reward_model
    struct:
    • name: style
      dtype: string
    • name: ground_truth
      dtype: string
  • name: extra_info
    struct:
    • name: index
      dtype: int64
  • name: dataset
    dtype: string
    splits:
  • name: train
    num_bytes: 78462192
    num_examples: 95373
    download_size: 75123456
    dataset_size: 78462192
    configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*

IFBench-VERL: Instruction Following Evaluation Dataset for VERL Training

Overview

IFBench-VERL is a comprehensive instruction-following evaluation dataset formatted for VERL (Versatile Reinforcement Learning) training pipelines. This dataset contains 95,373 high-quality examples with 54 different constraint types, enabling systematic training and evaluation of instruction-following capabilities in language models.

The dataset is converted from allenai/IF_multi_constraints_upto5 into VERL-compatible format with integrated IFEval-based reward modeling for automatic constraint verification.

Key Statistics

• Total Examples: 95,373
• Format: Parquet (VERL-compatible)
• Data Source: allenai/IF_multi_constraints_upto5
• License: ODC-BY-1.0 (Open Data Commons Attribution License)
• Conversion Date: November 3, 2025
• Constraint Types: 54 different instruction-following constraints
• Constraint Categories: 13 (keywords, language, length, content, format, punctuation, etc.)
• Max Constraints per Example: Up to 5 constraints
• Reward Model: IFEval-based automatic evaluation

Dataset Structure

Schema

The dataset follows the VERL standard schema format:

{
    'data_source': 'allenai/IF_multi_constraints_upto5',
    'prompt': [
        {
            'role': 'system',
            'content': 'You are a helpful assistant...'
        },
        {
            'role': 'user',
            'content': 'Write a paragraph about...'
        }
    ],
    'ability': 'instruction_following',
    'reward_model': {
        'style': 'ifeval',
        'ground_truth': '[{"instruction_id": ["keywords:existence"], "kwargs": [{"keywords": ["example", "test"]}]}]'
    },
    'extra_info': {
        'index': 0
    },
    'dataset': 'ifeval'
}

Field Descriptions

Instruction Following Tasks

This dataset tests 54 different constraint types across 13 major categories:

Constraint Categories

1. Keywords (8 constraints)

   • keywords:existence - Require specific keywords
   • keywords:frequency - Control keyword frequency
   • keywords:forbidden_words - Prohibit certain words
   • keywords:letter_frequency - Control letter frequency
   • keywords:no_adjacent_consecutive - Prevent adjacent letters
   • keywords:word_once - Require words exactly once
   • keywords:palindrome - Require palindrome words
   • keywords:keyword_specific_position - Place keywords at specific positions
   • keywords:start_end - Require start/end with specific text

2. Language (1 constraint)

   • language:response_language - Require response in specific language

3. Length Constraints (4 constraints)

   • length_constraints:number_sentences - Control sentence count
   • length_constraints:number_paragraphs - Control paragraph count
   • length_constraints:number_words - Control word count
   • length_constraints:nth_paragraph_first_word - Control specific paragraph's first word

4. Detectable Content (2 constraints)

   • detectable_content:number_placeholders - Require placeholders (e.g., [NAME])
   • detectable_content:postscript - Require postscript section

5. Detectable Format (8 constraints)

   • detectable_format:number_bullet_lists - Require bullet lists
   • detectable_format:constrained_response - Constrain response format
   • detectable_format:number_highlighted_sections - Require highlighted sections
   • detectable_format:multiple_sections - Require multiple sections
   • detectable_format:json_format - Require JSON format
   • detectable_format:title - Require title
   • detectable_format:sentence_hyphens - Require sentence with hyphens
   • detectable_format:square_brackets - Require square brackets
   • detectable_format:bigram_wrapping - Require bigram wrapping

6. Combination (2 constraints)

   • combination:two_responses - Require two separate responses
   • combination:repeat_prompt - Repeat prompt then answer

7. Start/End (2 constraints)

   • startend:end_checker - Control response ending
   • startend:quotation - Wrap response in quotation marks

8. Change Case (3 constraints)

   • change_case:capital_word_frequency - Control capitalized word frequency
   • change_case:english_capital - All uppercase
   • change_case:english_lowercase - All lowercase

9. Punctuation (4 constraints)

   • punctuation:no_comma - Prohibit commas
   • punctuation:punctuation_dot - End sentences with periods
   • punctuation:punctuation_exclamation - Use exclamation marks

10. Copy (4 constraints)

    • copy:repeat_phrase - Repeat specific phrase
    • copy:copy - Copy from prompt
    • copy:copying_simple - Simple copying task
    • copy:copying_multiple - Multiple copying operations

11. Paragraphs (2 constraints)

    • paragraphs:paragraphs - Basic paragraph structure
    • paragraphs:paragraphs2 - Alternative paragraph structure

12. First/Last Word (4 constraints)

    • first_word:first_word_sent - Control first word of sentence
    • first_word:first_word_answer - Control first word of answer
    • last_word:last_word_sent - Control last word of sentence
    • last_word:last_word_answer - Control last word of answer

13. Count (6 constraints)

    • count:lowercase_counting - Count lowercase letters
    • count:counting_composition - Counting composition
    • count:count_unique - Count unique elements
    • count:count_increment_word - Count with incremental words
    • letters:letter_counting - Count specific letters
    • letters:letter_counting2 - Alternative letter counting

Reward Model

IFEval-Based Automatic Evaluation

The dataset uses IFEval (Instruction Following Evaluation) for automatic constraint verification. Each example includes:

• Constraint Specification: JSON-encoded list of constraints with parameters
• Automatic Checking: Self-contained scorer in the datatrove library
• Binary Scoring: Each constraint is either satisfied (1.0) or not (0.0)
• Average Score: Final score is the average across all constraints

Ground Truth Format

The reward_model.ground_truth field contains JSON-encoded constraint specifications:

[
  {
    "instruction_id": [
      "keywords:existence",
      "length_constraints:number_words"
    ],
    "kwargs": [
      {"keywords": ["example", "test"]},
      {"num_words": 100, "relation": "at least"}
    ]
  }
]

Self-Contained Scorer

The IFEval scorer is included in the datatrove library at datatrove.utils.reward_score.ifeval:

from datatrove.utils.reward_score.ifeval import compute_score

# Evaluate model output against constraints
result = compute_score(
    model_output="Your model's response...",
    ground_truth='[{"instruction_id": [...], "kwargs": [...]}]'
)

# Result format
{
    "score": 0.85,  # Average success rate across all constraints
    "reward_fmt": 1.0,
    "reward_think": 1.0
}

Data Access

Using HuggingFace Datasets

from datasets import load_dataset

# Load the full dataset
dataset = load_dataset("sungyub/ifbench-verl", split="train")

# Example usage
example = dataset[0]
print(f"Prompt: {example['prompt']}")
print(f"Constraints: {example['reward_model']['ground_truth']}")

Using Pandas

import pandas as pd

# Load as pandas DataFrame
df = pd.read_parquet("hf://datasets/sungyub/ifbench-verl/train-00000-of-00001.parquet")

# View statistics
print(df.info())
print(df.head())

Using Polars

import polars as pl

# Load with Polars for faster processing
df = pl.read_parquet("hf://datasets/sungyub/ifbench-verl/train-00000-of-00001.parquet")

print(df.describe())

Technical Details

Metadata Standards

This dataset includes metadata following:

• ML Commons Croissant 1.1: Structured dataset metadata
• VERL Format: Compatible with VERL training pipelines
• IFEval Integration: Self-contained constraint checking

Supported Libraries

• datasets (HuggingFace)
• pandas
• polars
• dask

Dependencies for Reward Scoring

To use the IFEval reward scorer, install:

pip install datatrove[reward_scoring]

# Or install specific dependencies:
pip install langdetect immutabledict nltk

Use Cases

1. Instruction Following Training

Train language models to better follow complex instructions with multiple constraints:

from datasets import load_dataset

dataset = load_dataset("sungyub/ifbench-verl", split="train")

# Use for supervised fine-tuning or RL training
for example in dataset:
    prompt = example['prompt']
    constraints = example['reward_model']['ground_truth']
    # Train your model...

2. VERL/GRPO Training

Direct integration with VERL training pipelines:

# The 'dataset' field enables automatic verifier routing in GRPO
# The reward model automatically uses IFEval scoring

3. Benchmark Evaluation

Evaluate model instruction-following capabilities across 54 constraint types:

from datatrove.utils.reward_score.ifeval import compute_score

# Evaluate model output
score = compute_score(
    model_output=model_response,
    ground_truth=example['reward_model']['ground_truth']
)

print(f"Instruction following score: {score['score']:.2%}")

4. Constraint Analysis

Analyze which constraint types are most challenging:

import json

# Group by constraint types
constraint_stats = {}
for example in dataset:
    gt = json.loads(example['reward_model']['ground_truth'])
    for constraint_id in gt[0]['instruction_id']:
        category = constraint_id.split(':')[0]
        constraint_stats[category] = constraint_stats.get(category, 0) + 1

print(constraint_stats)

Citation and License

License

This dataset is licensed under ODC-BY-1.0 (Open Data Commons Attribution License), following the original dataset's license terms. You are free to:

• Share and redistribute the dataset
• Adapt and modify the dataset
• Use the dataset commercially

Under the condition of:

• Attribution: You must give appropriate credit to the original dataset creators

Citation

If you use this dataset, please cite the original IF dataset:

@misc{if_multi_constraints_upto5,
  title={IF Multi-Constraints (Up to 5)},
  author={AI2},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/allenai/IF_multi_constraints_upto5},
  year={2024}
}

And this VERL-formatted version:

@misc{ifbench_verl,
  title={IFBench-VERL: Instruction Following Evaluation Dataset for VERL Training},
  author={Sungyub Kim},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/sungyub/ifbench-verl},
  year={2025}
}

Responsible Use

This dataset is intended for research and educational purposes in accordance with:

• AI2's Responsible Use Guidelines
• Open Data Commons Attribution License terms

Please ensure ethical use of this dataset and be mindful of potential biases in instruction-following evaluation.

Additional Resources

• Original Dataset: allenai/IF_multi_constraints_upto5
• IFEval Paper: Instruction-Following Evaluation
• VERL Documentation: VERL Training Guide
• Datatrove Library: HuggingFace Datatrove

Contact

For questions or issues regarding this dataset conversion, please open an issue on the dataset repository.