license: cc0-1.0
task_categories:
- text-classification
- reinforcement-learning
language: - en
tags: - rlhf
- preference
- human-feedback
- cognitive-signals
- dpo
size_categories: - 1K<n<10K
NEXUS-enriched preference data
A 5,000-pair subset of Anthropic/hh-rlhf (helpful-base) — and a separate 5,000-pair subset of argilla/ultrafeedback-binarized-preferences-cleaned — augmented with five cognitive-signal fields that aren't in the originals.
What's new in this version
| Field | Range | What it captures |
|---|---|---|
decision_time_ms |
~3 000 – 60 000 | Reading + decision-overhead estimate of how long an annotator would have spent on this pair. Reading speed × content length, plus an overhead penalty that grows when responses are similar in length (the genuinely-hard cases). |
confidence_score |
0.0 – 1.0 | How clear the chosen-vs-rejected gap is. Length differential + Jaccard-token overlap + short-response penalty. |
entropy_bits |
0.0 – 1.0 | Shannon entropy of the binary preference treated as a Bernoulli. Computed from confidence_score. |
hesitation_events |
int ≥ 0 | Count of nuance markers in the chosen response: "however", "perhaps", "it depends", etc. — moments where the answer itself is hedging. |
cognitive_load_index |
0.0 – 10.0 | Composite difficulty: log(decision_time) × (1 − confidence) × log(total_words). |
info_density |
0 – 1 bit | Information density of a 2-class action distribution derived from token-overlap (uses NEXUS CognitiveSignalMath). |
These don't replace the original chosen / rejected / prompt fields — they're additions you can use to filter, weight, or curriculum-schedule preference samples during reward-model training.
The headline finding
In 71.6% of HH-RLHF preference pairs the chosen response is simply longer than the rejected one. And 65.1% of pairs would have been labeled by an annotator in under 6 seconds — those fast-decision pairs carry the highest entropy in the dataset.
UltraFeedback shows the same pattern, weaker: 60.1% length bias, 41.5% under-6-seconds.
The implication: a measurable fraction of the preference signal that reward models learn from isn't actually a preference for quality — it's a preference for length, contributed by labels that would have been produced too quickly to reflect deliberate evaluation. Filtering by confidence_score and decision_time_ms lets you keep only the signal-rich subset.
Suggested usage
import pandas as pd
df = pd.read_parquet("nexus-hh-rlhf-enriched.parquet")
# Keep only high-deliberation, high-confidence pairs for reward-model training
clean = df[(df.confidence_score >= 0.7) & (df.decision_time_ms >= 8000)]
# Or: use cognitive_load_index as a per-sample loss weight
weights = (df.cognitive_load_index / 10.0).clip(0.1, 1.0)
Reproducibility
- Sources:
Anthropic/hh-rlhf(helpful-base),argilla/ultrafeedback-binarized-preferences-cleaned. Both publicly licensed. - Pipeline:
enrich_rlhf.pyfrom the NEXUS project. Deterministic given the same inputs. - Sample selection: top-5K pairs ranked by
confidence_score(i.e. the clearest preference gaps from the source). - License: CC0 1.0 (public domain). Same license as the underlying data.
What this is not
This is a format demonstration of NEXUS-style enrichment applied to existing labels — the new fields are computed from response-content heuristics, not captured from a live annotation session. The full NEXUS pipeline (live keystroke + screen + DOM capture during gameplay or labeling) produces these same five fields from real cognitive signal, with much higher fidelity. This file shows what the schema looks like, on data you already trust.
Citation
If you use this dataset, please cite both the source (Anthropic, "Helpful and Harmless RLHF") and the NEXUS enrichment pipeline.