license: mit
task_categories:

• reinforcement-learning
• text-generation
  language:
• en
  tags:
• rlhf
• preference-learning
• human-feedback
• alignment
• stanford-cs217
• anthropic
• hh-rlhf
  size_categories:
• 1K<n<10K

CS217 Fixed HH-RLHF Dataset

Fixed subset of Anthropic's HH-RLHF dataset for reproducible RLHF experiments

Created for Stanford CS217: Hardware Accelerators for Machine Learning - Final Project

🔗 GitHub Repository: CS217-Final-Project

Dataset Description

This is a fixed subset of the Anthropic/hh-rlhf dataset, created to ensure reproducible experiments across all runs. The dataset contains human preference pairs for training reward models and RLHF (Reinforcement Learning from Human Feedback) algorithms.

Why This Dataset?

• ✅ Reproducibility: Fixed seed (42) ensures identical data across all experiments
• ✅ Proper Evaluation: Test samples from separate test split (no data leakage)
• ✅ Manageable Size: 1,200 samples ideal for rapid prototyping and energy benchmarking
• ✅ Standard Format: Compatible with HuggingFace TRL library and PPO trainers

Dataset Statistics

• Random Seed: 42
• Created: February 2026
• Format: Parquet
• Size: ~1.35 MB

Data Format

Each example contains two fields:

• chosen: The preferred (higher quality) assistant response
• rejected: The less preferred (lower quality) assistant response

Example Structure

{
  'chosen': '\n\nHuman: Why did cells originally combine together to create life?\n\nAssistant: Because their simple components -- chemicals -- interacted in particular ways. And because of chemical processes involving energy and replication, cells became able to reproduce themselves...',

  'rejected': '\n\nHuman: Why did cells originally combine together to create life?\n\nAssistant: Cells combine because they benefit from cooperation, since they can have less competition for resources by working together...'
}

Usage

Loading the Dataset

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("hivamoh/cs217-rlhf-dataset")

# Access splits
train_data = dataset['train']
test_data = dataset['test']

print(f"Train examples: {len(train_data)}")
print(f"Test examples: {len(test_data)}")

Using with TRL for RLHF Training

from datasets import load_dataset
from transformers import AutoTokenizer
from trl import PPOConfig, PPOTrainer

# Load dataset
dataset = load_dataset("hivamoh/cs217-rlhf-dataset")

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

# Extract prompts for PPO training
def extract_prompt(example):
    # Extract the human query from the chosen response
    text = example['chosen']
    if '\n\nHuman:' in text:
        prompt = text.split('\n\nAssistant:')[0] + '\n\nAssistant:'
        return {'query': prompt}
    return {'query': ''}

train_dataset = dataset['train'].map(extract_prompt)

# Configure PPO
ppo_config = PPOConfig(
    batch_size=8,
    learning_rate=1.41e-5,
    ...
)

# Train with PPOTrainer
# See GitHub repo for complete training script

Data Fields

• chosen (string): The preferred response in the preference pair
• rejected (string): The less preferred response in the preference pair

Both fields contain full conversation context including:

• Human prompts/questions
• Assistant responses
• Multi-turn dialogues

Dataset Creation

Source Dataset

This dataset is derived from Anthropic/hh-rlhf:

• Original train split: 160,800 examples
• Original test split: 8,552 examples

Selection Process

# Training samples
full_train = load_dataset('Anthropic/hh-rlhf', split='train')
train_subset = full_train.shuffle(seed=42).select(range(1000))

# Test samples
full_test = load_dataset('Anthropic/hh-rlhf', split='test')
test_subset = full_test.shuffle(seed=42).select(range(200))

Creation Script

The dataset was created using create_fixed_dataset.py from the CS217 project repository.

To reproduce:

git clone https://github.com/HivaMohammadzadeh1/CS217-Final-Project.git
cd CS217-Final-Project
python baseline_energy/create_fixed_dataset.py \
  --train-size 1000 \
  --test-size 200 \
  --seed 42

Use Cases

This dataset is designed for:

1. RLHF Research: Training and evaluating reward models and RLHF algorithms
2. Energy Benchmarking: Fixed dataset for measuring energy consumption across runs
3. Hardware Acceleration: Evaluating quantization policies and FPGA implementations
4. Reproducibility Studies: Comparing different approaches on identical data
5. Educational Purposes: Learning RLHF without downloading full 160K+ dataset

Project Context

This dataset is part of a Stanford CS217 final project investigating:

"Layer-Adaptive MX Format Quantization for RLHF Energy Optimization on FPGAs"

Research Question

Can layer-adaptive Microscaling (MX) format quantization (MXFP4/MXFP8) reduce RLHF training energy consumption on FPGA hardware compared to GPU baseline, without degrading alignment quality?

Key Components

• Model: Qwen2.5-0.5B-Instruct (494M parameters)
• Algorithm: PPO (Proximal Policy Optimization)
• Hardware: AWS GPU (T4) baseline, AWS FPGA (F2) target
• Quantization: MXFP4/MXFP8 with layer-adaptive precision

Citation

If you use this dataset, please cite:

@misc{cs217-rlhf-dataset,
  title={CS217 Fixed HH-RLHF Dataset},
  author={Zaad, Hiva and Griffith, Grant and Adkins, Daniel},
  year={2026},
  publisher={HuggingFace},
  howpublished={\url{https://huggingface.co/datasets/hivamoh/cs217-rlhf-dataset}}
}

And the original Anthropic HH-RLHF dataset:

@article{bai2022training,
  title={Training a helpful and harmless assistant with reinforcement learning from human feedback},
  author={Bai, Yuntao and others},
  journal={arXiv preprint arXiv:2204.05862},
  year={2022}
}

License

MIT License - Same as original Anthropic/hh-rlhf dataset

Links

• 📁 GitHub Repository: HivaMohammadzadeh1/CS217-Final-Project
• 📊 Original Dataset: Anthropic/hh-rlhf
• 🤗 Model: Qwen/Qwen2.5-0.5B-Instruct
• 📖 TRL Library: HuggingFace TRL

Contact

For questions or issues:

• Open an issue on GitHub
• Course: Stanford CS217 - Hardware Accelerators for Machine Learning

Acknowledgments

• Anthropic for the original HH-RLHF dataset
• HuggingFace for hosting and tools
• Stanford CS217 course staff
• Project team: Hiva Zaad, Grant Griffith, Daniel Adkins