task_categories:

• question-answering
  language:
• en
  size_categories:
• 10K<n<100K

RMB Dataset

A comprehensive benchmark dataset for evaluating reward models in large language model alignment, containing pairwise preference comparisons and best-of-N evaluations.

Overview

The RMB (Reward Model Benchmark) dataset is designed to comprehensively benchmark reward models in LLM alignment. It contains preference data across multiple dimensions including harmlessness and helpfulness, with various task types and response styles.

Citation

If you use this dataset in your research, please cite:

@misc{zhou2025rmbcomprehensivelybenchmarkingreward,
    title={RMB: Comprehensively Benchmarking Reward Models in LLM Alignment}, 
    author={Enyu Zhou and Guodong Zheng and Binghai Wang and Zhiheng Xi and Shihan Dou and Rong Bao and Wei Shen and Limao Xiong and Jessica Fan and Yurong Mou and Rui Zheng and Tao Gui and Qi Zhang and Xuanjing Huang},
    year={2025},
    eprint={2410.09893},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2410.09893}, 
}

Dataset Structure

The dataset is available on Hugging Face Hub at: dd101bb/RMB_dataset

Splits

The dataset contains two main splits:

1. pairwise: Pairwise preference comparisons (chosen vs rejected responses)
2. bon: Best-of-N evaluations (best response vs multiple loser responses)

Data Schema

Each example in the dataset follows a unified schema:

{
    "uid": str,                    # Unique identifier
    "conversation": [               # List of conversation messages
        {
            "role": str,            # "user" or "assistant"
            "content": str,         # Message content
            "language": str         # Language of the message
        },
        ...
    ],
    "responses": [                  # List of response candidates
        {
            "llm_name": str,        # Name of the model that generated the response
            "answer": str           # Response text
        },
        ...
    ],
    "preferred_index": int,         # Index of the preferred response in responses list
    "category_path": str,           # Category path (e.g., "Pairwise_set/Harmlessness/S1")
    "split_type": str               # "pairwise" or "bon"
}

Categories

The dataset covers multiple categories:

• Harmlessness: Safety evaluations across different scenarios (S1-S11, multi)
• Helpfulness: Various task types including:
  • Brainstorming
  • Chat (Casual Conversation, Discussion, Supportive Conversation)
  • Classification
  • Closed QA
  • Code
  • Generation
  • Open QA
  • Reasoning
  • Rewrite
  • Role Playing
  • Summarization
  • Translation

Usage

from datasets import load_dataset

# Load the pairwise split
dataset = load_dataset("dd101bb/RMB_dataset", split="pairwise")

# Or load both splits
dataset_dict = load_dataset("dd101bb/RMB_dataset")
pairwise_data = dataset_dict["pairwise"]
bon_data = dataset_dict["bon"]

Dataset Statistics

• Pairwise Split: Contains pairwise preference comparisons
• BoN Split: Contains best-of-N evaluations
• Categories: Covers both harmlessness and helpfulness dimensions
• Task Types: Multiple task types within helpfulness category
• Languages: Primarily English, with language metadata included

Data Format Details

Pairwise Examples

For pairwise examples, each entry contains:

• A conversation (user input)
• Two responses: one chosen (preferred) and one rejected
• The preferred_index is always 0 (first response is chosen)
• The rejected response is at index 1

Best-of-N Examples

For BoN examples, each entry contains:

• A conversation (user input)
• Multiple responses: one best response and a list of loser responses
• The preferred_index is always 0 (first response is best)
• The losers list contains all other responses

License

Please refer to the original paper and dataset repository for license information.

References

• Paper: RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
• Hugging Face: dd101bb/RMB_dataset

Contributing

If you find any issues with the dataset or have suggestions for improvements, please open an issue or submit a pull request.