license: apache-2.0
task_categories:

• text-classification
  language:
• en
  tags:
• reward
• evaluation
• rlhf
  size_categories:
• 10K<n<100K

Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models

Introduction

We introduce MRMBench, a comprehensive benchmark for evaluating reward models across six key preference dimensions: harmlessness, helpfulness, correctness, coherence, complexity, and verbosity. Unlike traditional pairwise-ranking evaluations that reduce preference learning to a single binary choice, MRMBench probes whether reward models genuinely encode rich, multi-dimensional human preferences. We further propose inference-time probing, a simple yet effective analysis method that identifies which preference dimensions a reward model relies on during prediction, enhancing interpretability and revealing potential failure modes. Our paper is available at this link.

Overview

The data statistics are as follows:

File hierachy

data
├─ mrm_easy
│    ├─ coherence
│    │    ├─ test.json
│    │    └─ train.json
│    ├─ complexity
│    │   ... ...
... ...
│    └─ verbosity
│        ... ...
└─ mrm_hard
       ├─ coherence
       │    ├─ test.json
       │    └─ train.json
    ... ...

Description

In the MRMBench evaluation pipeline:

1. A reward model is trained on the train.json file for a given task.
2. The model's accuracy is then evaluated on the corresponding test.json file.

In each .json file, the format of data is:

[
    {
        "input": "explain master slave replication nsql",
        "output": "In the context of NoSQL ... ...",
        "label": 1
    },
    ......
        {
        "input": "generate prompts that ... ...",
        "output": "Sure, here are some prompts ... ...",
        "label": 0
    },
    ......
]

The full fine-tuning and evaluation scripts are available in the official MRMBench repository.

Evaluation Results

We present a subset of evaluation results for several reward models on MRMBench. Additional results can be found in our full paper.

MRMBench-Easy

MRMBench-Hard

Citation

@misc{wang2025probingpreferencerepresentationsmultidimensional,
      title={Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models}, 
      author={Chenglong Wang and Yifu Huo and Yang Gan and Yongyu Mu and Qiaozhi He and Murun Yang and Bei Li and Chunliang Zhang and Tongran Liu and Anxiang Ma and Zhengtao Yu and Jingbo Zhu and Tong Xiao},
      year={2025},
      eprint={2511.12464},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2511.12464}, 
}

Acknowledgement

We thank the contributions of the following papers:

[1] Wang, Zhilin, et al. "Helpsteer: Multi-attribute helpfulness dataset for steerlm." Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.  
[2] Wang, Zhilin, et al. "Helpsteer 2: Open-source dataset for training top-performing reward models." Advances in Neural Information Processing Systems 37 (2024): 1474-1501.  
[3] Ji, Jiaming, et al. "Pku-saferlhf: Towards multi-level safety alignment for llms with human preference." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.