license: mit
language:
- en
task_categories: - text-generation
- reinforcement-learning
tags: - dpo
- preference-optimization
- rlhf
- tiny-models
- edge-ai
- alignment
pretty_name: HH-RLHF Helpful DPO Preference Pairs (10k)
size_categories: - 10K<n<100K
HH-RLHF Helpful DPO Preference Pairs · 10k
10,000 real human preference pairs for teaching a tiny language model (≤50M params) what a good assistant sounds like — more helpful, more natural, less evasive.
-0FB5AE){kind=icon}
Why this dataset exists
This is the preference-tuning stage of an end-to-end tiny-model training pipeline:
Pretraining ──► SFT ──► DPO (this dataset) ──► Tiny Edge Assistant
After SFT teaches the model how to speak, this dataset teaches it which way of speaking is better.
Every row is a real human preference judgment from the Anthropic HH-RLHF collection:
- chosen — the response a good, helpful support assistant should sound like
- rejected — the evasive, unhelpful, or robotic responses to avoid
This is the first step of a longer journey: squeezing as much intelligence as possible out of tiny models that can run on edge devices — phones, microcontrollers, and offline hardware.
Dataset description
| Source | Anthropic/hh-rlhf — data_dir="helpful-base", split="train" |
| Sampling | shuffle(seed=42) → first 10,000 valid pairs |
| Language | English only |
| Columns | prompt, chosen, rejected |
| Format | Parquet (train.parquet) |
Column layout
| Column | Description |
|---|---|
prompt |
Full conversation history up to the final Assistant: turn (Human/Assistant dialogue) |
chosen |
The assistant reply a human preferred |
rejected |
The assistant reply a human did not prefer |
Corpus statistics
| Metric | Prompt | Chosen | Rejected |
|---|---|---|---|
| Avg length (words) | 92.9 | 44.1 | 34.4 |
| Max length (words) | 753 | 540 | 292 |
Usage
from datasets import load_dataset
from trl import DPOTrainer
ds = load_dataset("salisai/hh-rlhf-helpful-dpo-10k", split="train")
# rows are already DPO-ready:
# ds[0] = {"prompt": ..., "chosen": ..., "rejected": ...}
You can pass it straight into DPOTrainer with:
DPOTrainer(
model=model, # your SFT-finished tiny model (≤50M)
train_dataset=ds,
beta=0.1,
...
)
Sample row
Prompt
Human: When was Pizza Hut founded?
Assistant: Hm... Pizza Hut was founded in 1958.
Human: What was the founders name?
Chosen (preferred)
Oh sorry, my research doesn't have that. I was just able to get the founder of the company who also opened it.
Rejected (avoid)
Can you tell me what you're looking for? Does "the founder's name" mean that you want to know the name of the founder of the chain of restaurants that operates in many places today? Or maybe you...
Roadmap: tiny models on the edge
| Stage | Dataset | Goal |
|---|---|---|
| 1. Pretraining | token streams | raw language ability |
| 2. SFT | instruction pairs | learn to answer |
| 3. DPO (this) | hh-rlhf helpful pairs | learn how to answer well |
| 4. Deployment | — | run on edge devices |
Everything here is deliberately small and English-only so a ≤50M-param model can actually absorb the signal. The end goal: useful, natural, non-evasive assistants running on resource-constrained edge hardware — this 10k preference set is the alignment step that makes them sound human instead of robotic.
Source & license
- Source dataset: Anthropic/hh-rlhf — Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (Bai et al., 2022)
- Built from: the
helpful-basesubset, human preference labels collected by Anthropic. - This repository: extracted, cleaned (chosen/rejected prompts validated to match), and formatted into DPO-ready rows.
- License:
mit