Scalar RM Per-Axis Blindness Experiment
Controlled experiment testing whether scalar reward models have measurable per-axis blindness on IFEval multi-constraint prompts, and whether this blindness predicts behavior under best-of-N selection.
Quick Start (Google Colab)
- Open Google Colab
- Change runtime to A100 GPU (Runtime → Change runtime type → A100)
- Set your HF token in Colab Secrets (🔑 icon on the left sidebar): add
HF_TOKENwith your token - Run this in a cell:
!wget https://huggingface.co/datasets/abhi14/rm-axis-blindness-experiment/resolve/main/colab_experiment.py
!python colab_experiment.py
Estimated runtime: 2-3 hours on A100.
What it does
Hypothesis
Tests two claims about scalar reward model "blindness" on specific utility axes:
- STRONG CLAIM: Axes where coverage < 0.7 will have BoN delta ≤ 0
- WEAK CLAIM: BoN delta is monotonically increasing in coverage with positive slope, after controlling for length
Pipeline (9 stages)
- Load IFEval, filter to multi-constraint prompts (≥3 constraints)
- Map 25 instruction types to 5 axes: length, format, content, case, combination
- Generate 32 completions per prompt with 2 generators (Qwen-2.5-7B + Mistral-7B)
- Evaluate all completions with IFEval's official deterministic checkers
- Score all completions with 3 architecturally diverse reward models
- Compute per-axis coverage (loose + strict)
- Compute best-of-N delta with correct expected baseline
- Run full statistical suite (OLS, mixed-effects, leave-one-out, length-bias probe)
- Generate all deliverables
Models used
- Generators: Qwen/Qwen2.5-7B-Instruct, mistralai/Mistral-7B-Instruct-v0.3
- Reward Models:
- Skywork/Skywork-Reward-V2-Llama-3.1-8B (Llama)
- Ray2333/reward-model-Mistral-7B-instruct-Unified-Feedback (Mistral)
- OpenAssistant/reward-model-deberta-v3-large-v2 (DeBERTa)
Deliverables (12 files)
| File | Description |
|---|---|
coverage_heatmap_strict.png |
Strict coverage heatmap: RMs × axes |
coverage_heatmap_loose.png |
Loose coverage heatmap |
bon_delta_heatmap_n32.png |
BoN delta heatmap per generator |
coverage_vs_delta_scatter.png |
Coverage vs delta scatter with OLS fit |
bon_n_scaling.png |
BoN delta scaling with N=8,16,32 |
robustness_table.md |
Statistical robustness results |
report.md |
Final report with verdict on both claims |
raw_scores.jsonl |
Per-completion scores and checker results |
rm_scores_raw.json |
Raw RM score dump |
checker_errors.log |
IFEval checker error log |
data_quality.json |
Template detection statistics |
benchmark_info.json |
Dataset statistics |