license: mit
language:

• en
  size_categories:
• 100K<n<1M
  task_categories:
• text-generation
  tags:
• dpo
• rlhf
• preference
• trl
• hh-rlhf
• anthropic
  configs:
• config_name: default
  data_files:
  • split: train
    path: data/train.parquet
  • split: test
    path: data/test.parquet

hh-rlhf-dpo-native

Anthropic/hh-rlhf in the native TRL conversational-preference format, with zero-gradient
and unparsable pairs removed.

The original dataset ships two raw strings (chosen, rejected) containing the entire
conversation serialized with \n\nHuman: / \n\nAssistant: separators. Every user has to
write their own parser, and that parser has to make a judgement call on ~2% of rows that are
corrupted. This release does that work once, deterministically, and publishes the result as
three typed list-of-messages columns.

from datasets import load_dataset
from trl import DPOTrainer

ds = load_dataset("Archangel-system/hh-rlhf-dpo-native")
trainer = DPOTrainer(model=model, args=cfg, train_dataset=ds["train"], processing_class=tok)

No parsing. No .map(). That is the acceptance criterion for this dataset, and it is tested
(see acceptance_dpo.py — a real DPOTrainer step runs on these files).

Schema

subset is recovered provenance: the HF auto-converted parquet of the source flattens the
four original directories into one unlabelled split. We re-derived the label by concatenating the
four original .jsonl.gz shards and verifying byte-for-byte that the concatenation reproduces the
flattened parquet row-for-row, in order (asserted in build.py, not assumed).

Size

Overall retention: 97.9% train / 98.0% test.

What was removed, and why

Every row is charged to the first reason that fires, so the counters sum exactly to the number
of dropped rows. Nothing was repaired by rewriting: an ambiguous row is dropped with a reason,
never guessed.

The two that matter

740 zero-gradient pairs (0.46% of train). chosen == rejected byte for byte. In DPO the
implicit reward margin is identically zero, so these contribute exactly nothing to the gradient
while still consuming batch slots. A further 434 are identical under normalization — same
pathology, invisible to a byte comparison.

1,308 rows with a role marker inside the completion (0.81% of train) — the largest defect, and
the one a naive split("\n\nAssistant: ") silently swallows. The source text carries a corrupted
turn boundary, e.g. a completion whose content literally begins Human: A police station can certainly be.... A parser that ignores this trains the model to emit Human: and role-play the
user. Two source rows use a no-trailing-space \n\nAssistant: separator, which is why detection
uses a looser regex than splitting does.

Decontamination

Train/test overlap is checked under an aggressive normalization — re.sub(r"[^a-z0-9]+", "", s.lower()) —
applied by the same function that performs dedup and by verify.py. A purge norm weaker than the
control norm leaves residue.

• 0 train rows shared a full (prompt, chosen, rejected) triple with test.
• 182 train rows shared a test prompt and were removed. Purging is train-side only; the eval
  split is small and sacred, and trimming it would destroy the thing being measured.
• Final measured train↔test prompt overlap: 0.
• Duplicate triples within train after parsing: 0.

Note that 156,603 distinct prompts across 157,462 train rows is expected, not a defect: the source
deliberately contains several distinct preference pairs per prompt (notably in
helpful-rejection-sampled). Only exact triple duplicates are removed.

Length balance

DPO on length-biased pairs trains verbosity rather than quality, so the chosen − rejected
character delta is reported rather than hidden:

Near-balanced (a coin flip would be 50%), with a small residual bias toward longer chosen
inherited from the source. Worth a length-penalty check if your reward margin rises while
qualitative eval does not.

Prompt turns: median 3, p95 7, max 67 (train).

Subset distribution (train / test)

Reproducing

python build.py     # deterministic, seed 42, no network beyond the source download
python verify.py    # re-reads the PUBLISHED files from the Hub and asserts every claim above

build.py uses regex parsing only. No LLM is involved anywhere in the transformation — nothing
is rewritten, reformulated or invented. The build asserts its own guarantees before writing
(stats.json → self_checks, all zero).

File hashes (sha256):

• data/train.parquet — 9a6daf288f5688965be9e216758810d280949523ec6429422561b65334edd212
• data/test.parquet — 1664f7883a5b61e8190346de419ae991cb6f9a950d8bde04da54c6db0b846d72

Full counters: stats.json.

Limitations

• Content is unchanged. This is a reformatting and filtering pass. All the known properties of
  hh-rlhf remain: 2022-era crowdworker preferences, noisy and sometimes inconsistent labels,
  harmless-base contains explicitly harmful requests, and annotator agreement in the source is
  moderate. It is a format fix, not a quality uplift.
• Rows dropped as ambiguous are not recoverable from this release; use source_index against the
  source to inspect them.
• The subset label is derived positionally (verified byte-for-byte), not read from a source field.

License and attribution

MIT, inherited from the source.

Derived from Anthropic/hh-rlhf (MIT),
revision 09be8c5bbc57cb3887f3a9732ad6aa7ec602a1fa. This is a reformatted and filtered derivative:
the conversational content is entirely Anthropic's, unmodified; this work contributes the parsing,
the typed schema, the defect taxonomy and the decontamination.

@misc{bai2022training,
  title={Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback},
  author={Bai, Yuntao and others},
  year={2022},
  eprint={2204.05862},
  archivePrefix={arXiv}
}