license: odc-by
task_categories:

• text-generation
• reinforcement-learning
  language:
• en
  tags:
• evaluation
• ifeval
• instruction-following
• rlvr
• verl
• reasoning
  size_categories:
• 10K<n<100K
  pretty_name: RLVR-IFeval-VERL
  dataset_info:
  features:
  • name: data_source
    dtype: string
  • name: prompt
    list:
    • name: role
      dtype: string
    • name: content
      dtype: string
  • name: ability
    dtype: string
  • name: reward_model
    struct:
    • name: style
      dtype: string
    • name: ground_truth
      dtype: string
  • name: extra_info
    struct:
    • name: index
      dtype: int64
  • name: dataset
    dtype: string
    splits:
  • name: train
    num_bytes: 11309939
    num_examples: 14973
    download_size: 11309939
    dataset_size: 11309939
    configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*.parquet

RLVR-IFeval-VERL: Instruction Following Evaluation from RLVR

Overview

This dataset contains 14,973 instruction-following examples from the RLVR-IFeval dataset, transformed into the IFBench-VERL format for training reinforcement learning models with VERL.

The dataset provides structured evaluation data for testing language models' ability to follow explicit constraints and instructions, covering 25 distinct constraint types across multiple categories.

Dataset Statistics

Dataset Structure

Fields

Each example contains the following fields:

• data_source (string): Source dataset identifier (always "allenai/IF_multi_constraints_upto5")
• prompt (list): Conversation messages in chat format
  • role (string): Message role (typically "user")
  • content (string): Instruction text with embedded constraints
• ability (string): Task category (always "instruction_following")
• reward_model (dict): Reward model configuration for evaluation
  • style (string): Reward type (always "ifeval")
  • ground_truth (string): Python literal string encoding constraint specifications
• extra_info (dict): Additional metadata
  • index (int64): Original example index from source dataset
• dataset (string): Dataset identifier (always "ifeval")

Example

{
  "data_source": "allenai/IF_multi_constraints_upto5",
  "prompt": [
    {
      "role": "user",
      "content": "Describe the purpose of internet protocol version 6 (IPv6).\\n\\n Your entire response should be in English, and in all lowercase letters. No capital letters are allowed."
    }
  ],
  "ability": "instruction_following",
  "reward_model": {
    "style": "ifeval",
    "ground_truth": "[{'instruction_id': ['change_case:english_lowercase'], 'kwargs': [None]}]"
  },
  "extra_info": {
    "index": 0
  },
  "dataset": "ifeval"
}

Constraint Types

The dataset includes 25 distinct instruction-following constraint types, organized into 9 categories:

1. Keywords (4 types)

• verify_keywords → keywords:existence
  • Requires specific keywords to appear in the response
• verify_keyword_frequency → keywords:frequency
  • Requires a keyword to appear a specific number of times
• validate_forbidden_words → keywords:forbidden_words
  • Prohibits certain words from appearing
• verify_letter_frequency → keywords:letter_frequency
  • Requires a specific letter to appear a certain number of times

2. Language (1 type)

• validate_response_language → language:response_language
  • Requires the response to be in a specific language

3. Length Constraints (4 types)

• verify_paragraph_count → length_constraints:number_paragraphs
  • Requires a specific number of paragraphs
• validate_word_constraint → length_constraints:number_words
  • Requires a specific number of words (with quantifiers: "at least", "at most")
• verify_sentence_constraint → length_constraints:number_sentences
  • Requires a specific number of sentences
• validate_paragraphs → length_constraints:nth_paragraph_first_word
  • Requires the nth paragraph to start with a specific word

4. Detectable Content (2 types)

• verify_postscript → detectable_content:postscript
  • Requires a postscript (e.g., "P.S.") in the response
• validate_placeholders → detectable_content:number_placeholders
  • Requires a specific number of placeholders (e.g., [placeholder])

5. Detectable Format (6 types)

• verify_bullet_points → detectable_format:number_bullet_lists
  • Requires a specific number of bullet points
• validate_title → detectable_format:title
  • Requires the response to include a title
• validate_choice → detectable_format:constrained_response
  • Requires the response to be one of predefined options
• validate_highlighted_sections → detectable_format:number_highlighted_sections
  • Requires a specific number of highlighted sections (e.g., *text*)
• validate_sections → detectable_format:multiple_sections
  • Requires the response to be divided into named sections
• validate_json_format → detectable_format:json_format
  • Requires the response to be in valid JSON format

6. Combination (2 types)

• validate_repeat_prompt → combination:repeat_prompt
  • Requires the response to repeat the original prompt
• validate_two_responses → combination:two_responses
  • Requires two distinct responses separated by specific markers

7. Case Changes (3 types)

• validate_uppercase → change_case:english_capital
  • Requires all text to be uppercase
• validate_lowercase → change_case:english_lowercase
  • Requires all text to be lowercase
• validate_frequency_capital_words → change_case:capital_word_frequency
  • Requires a specific number of capitalized words

8. Start/End (2 types)

• validate_end → startend:end_checker
  • Requires the response to end with a specific phrase
• validate_quotation → startend:quotation
  • Requires the response to be wrapped in quotation marks

9. Punctuation (1 type)

• validate_no_commas → punctuation:no_comma
  • Prohibits the use of commas in the response

Usage

Loading the Dataset

from datasets import load_dataset

# Load the full dataset
dataset = load_dataset("sungyub/ifeval-rlvr-verl", split="train")

# Load a subset
dataset = load_dataset("sungyub/ifeval-rlvr-verl", split="train[:100]")

print(f"Loaded {len(dataset)} examples")
print(dataset[0])

Using with Datatrove Scorer

This dataset is compatible with the datatrove IFEval scorer:

from datatrove.utils.reward_score import compute_score

# Get an example
example = dataset[0]

# Generate a response (example using XML format)
response = """<think>
I need to write about IPv6 in all lowercase letters.
</think>

ipv6 is the successor to ipv4, providing a vastly larger address space with 128-bit addresses instead of 32-bit. this enables more devices to connect to the internet directly without requiring network address translation.
"""

# Compute score
score = compute_score(
    data_source="sungyub/ifeval-rlvr-verl",
    solution_str=response,
    ground_truth=example["reward_model"]["ground_truth"],
    format_type="auto"  # Supports both XML and GPT-OSS formats
)

print(score)
# Output: {'score': 1.0, 'reward_fmt': 1.0, 'reward_think': 1.0}

Using with VERL

The dataset can be directly used with VERL for reinforcement learning training:

from verl import DataProto

# Load dataset
dataset = load_dataset("sungyub/ifeval-rlvr-verl", split="train")

# Convert to VERL format
verl_data = []
for example in dataset:
    data_proto = DataProto(
        prompt=example["prompt"],
        ability=example["ability"],
        reward_model=example["reward_model"],
        extra_info=example["extra_info"],
    )
    verl_data.append(data_proto)

Response Format Support

The scorer supports two response formats:

1. XML Format (Traditional)

response = """<think>
Reasoning process here
</think>

Final response here
"""

2. GPT-OSS Format

response = """<|start|>assistant<|channel|>analysis<|message|>
Reasoning process here
<|end|>

<|start|>assistant<|channel|>final<|message|>
Final response here
<|return|>
"""

The format_type="auto" parameter automatically detects the format.

Data Transformation

This dataset was created by transforming the allenai/RLVR-IFeval dataset to match the IFBench-VERL schema. The transformation includes:

1. Function Mapping: All 25 RLVR constraint functions are mapped to existing IFEval instruction types
2. Parameter Conversion: RLVR parameter names are converted to IFEval parameter names (e.g., N → num_paragraphs)
3. Quantifier Normalization: RLVR quantifiers are normalized to IFEval relations:
   • "at most N" → "less than N+1"
   • "around N" → "at least N"
4. Ground Truth Format: Ground truth is stored as Python literal strings (using repr()) for compatibility with ast.literal_eval()

Evaluation

The dataset can be used to evaluate instruction-following capabilities by:

1. Generating responses to the prompts
2. Using the IFEval scorer to compute reward scores
3. Analyzing performance across different constraint types

Scores are returned as a dictionary with:

• score: Overall score (0.0 to 1.0)
• reward_fmt: Format reward (validates response structure)
• reward_think: Thinking reward (validates reasoning section)

Citation

If you use this dataset, please cite both the original RLVR-IFeval dataset and this transformation:

@misc{rlvr-ifeval,
  title={RLVR-IFeval: Instruction Following Evaluation Dataset},
  author={Allen Institute for AI},
  year={2024},
  url={https://huggingface.co/datasets/allenai/RLVR-IFeval}
}

@misc{ifeval-rlvr-verl,
  title={RLVR-IFeval-VERL: IFBench-VERL Format Transformation},
  author={Sungyub Kim},
  year={2025},
  url={https://huggingface.co/datasets/sungyub/ifeval-rlvr-verl}
}

For the original IFEval benchmark:

@misc{zhou2023instructionfollowing,
  title={Instruction-Following Evaluation for Large Language Models},
  author={Jeffrey Zhou and Tianjian Lu and Swaroop Mishra and Siddhartha Brahma and Sujoy Basu and Yi Luan and Denny Zhou and Le Hou},
  year={2023},
  eprint={2311.07911},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

License

This dataset is released under the ODC-BY license, consistent with the source dataset. You are free to:

• Share: Copy and redistribute the data
• Adapt: Transform and build upon the data
• Use commercially: Use the data for commercial purposes

Under the following terms:

• Attribution: You must give appropriate credit and indicate if changes were made

Related Resources

• Source Dataset: allenai/RLVR-IFeval
• Similar Dataset: sungyub/ifbench-verl (95K examples from multiple sources)
• Scorer Library: datatrove
• VERL Framework: VERL

Acknowledgments

• Allen Institute for AI for creating the original RLVR-IFeval dataset
• Google Research for the IFEval benchmark
• HuggingFace for the datatrove library
• Volcengine for the VERL framework