license: cc-by-sa-4.0
language:

• en
  task_categories:
• text-classification
• text-generation
  tags:
• moral-foundations
• preference
• dpo
• activation-steering
• morality
  size_categories:
• 10K<n<100K
  configs:
• config_name: pairs
  data_files: pairs.parquet
  default: true
• config_name: completions
  data_files: completions.parquet

Moral Stories, labelled by moral foundation

Why moral foundations? Moral Foundations Theory is one of the few maps of human values that is
calibrated against real people: it was drawn from cross-cultural survey studies and aims to hold
across societies, not just Western ones. That breadth is what matters here. If we want to study
how a constructed intelligence, a kind of moral alien, reasons about right and wrong, we need a
measure that generalises beyond any one culture. Moral foundations at least aims to span human cultures,
and ideally gives us a frame that reaches a little past them.

Why paired data? Preference and contrastive methods learn from the relative difference
between two responses: DPO and SimPO want a preferred and a dispreferred action, and activation
steering wants the difference vector between them. They work best when the two sides are matched,
same situation, differing only on the moral axis, so the signal is moral content and not some
confound. That is what this dataset provides, and the foundation tag lets you train one axis and
evaluate it on the same axes with tinymfv. (Plenty of moral
fine-tuning needs no pairs at all; this is for the methods that do.)

What. 12,000 matched pairs. Each row is a situation with a chosen action that observes a
social norm and a rejected action that violates it, tagged with the moral foundation(s) the two
actions differ on (Haidt's care, fairness, loyalty, authority, sanctity). The two sides are
length-matched (per-pair word-count Cohen's d about -0.1), so the contrast is moral content, not
length.

These are preference pairs: chosen is the moral action, rejected the immoral one, so the
rows drop straight into DPO / SimPO / ORPO or any pairwise-preference trainer, with the
foundation tag letting you select or stratify by moral axis.

Where it came from.

• Pairs: Moral Stories (Emelin et al. 2021), MIT.
• Foundation labels: Social Chemistry 101
  (Forbes et al. 2020), CC-BY-SA-4.0.
• Precedent for preference use: Histoires Morales
  (Leteno et al. 2025) translated Moral Stories to French and reformatted it for DPO.

What we did. Each row is a small moral dilemma with two endings: one that follows a social
rule, one that breaks it. We wanted every pair tagged with the moral foundation it turns on (care,
fairness, loyalty, authority, or sanctity) without hand-labeling twelve thousand of them.

We got the tags almost for free. Another dataset, Social Chemistry 101, had already collected
thousands of one-line moral rules, things like "It's wrong to bully people," and paid crowd
workers to tag each one with the moral foundation behind it. It turns out Moral Stories wrote its
dilemmas around those exact sentences, word for word. So we just line the two datasets up on the
shared sentence and copy the human tag across. About 94% of our pairs find their match.

Two caveats worth knowing. Most rules were tagged by a single worker, so for those the label is
really one person's opinion; where several people tagged the same rule we keep a foundation if most
of them agreed, and the raw vote shares stay in the frac_* columns if you want to set your own
bar. And Social Chemistry only covers five foundations, so Liberty and Social Norms are missing.

So treat these as a strong, human-sourced starting point rather than gospel. We checked them two
ways. First, against Social Chemistry's own 50-worker consensus set: a lone worker matches the
majority with Cohen's kappa 0.39-0.50, and an LLM judge separates the consensus foundation at
AUROC 0.62-0.79, so neither the human nor the machine label is gold, they are both noisy
classifiers of a fuzzy thing. Second, we re-judged every pair with an LLM. That gives the foundation
label, described next.

The recommended label: foundation

We wanted a label that says which moral foundation each pair actually turns on, and for most uses
the foundation column is the only one you need.

To build it, three LLMs, one each from three different labs (DeepSeek, Alibaba, Google), re-read
every pair and rated how much each action, the right one and the wrong one, violates each
foundation. The pair's signal is the gap between the two: how much more the rejected action
violates a foundation than the chosen one. The foundation with the largest gap is foundation.
We re-judged from scratch rather than reuse the crowd label because the crowd labeled the
underlying norm, not the actual story written around it, and the two sometimes describe different
situations (the journal has examples).

If you want more than the single label:

• llm_{care,fairness,loyalty,authority,sanctity,liberty}: the gap for each foundation, in points
  on the 1-5 violation scale (how much more violating the rejected action is than the chosen one,
  averaged over the three judges). A graded, multi-axis score; foundation is just its largest
  entry. We leave it on the raw scale rather than rescale it to a "fraction of people", since the
  only human anchor we have (the 50-worker set) is a weak, different-task fit.
• llm_std_*: how much the three labs disagreed on that score. Filter for small values when you
  want only the pairs they were confident about.
• human_agrees: whether the original crowd label includes foundation. True for 60% of
  human-labeled pairs; the rest are where the LLMs, reading the real pair, landed somewhere the
  crowd's norm-level label did not.

The crowd's own labels (frac_*, foundations, ...) stay in the table if you want to compare or
fall back to them.

How much to trust it: the three labs pick the same top foundation 76-79% of the time, while two
random human annotators agree on the exact set only 27% of the time, so the machine label is the
more reproducible of the two (not necessarily the more correct). Social Norms was used only as a
control and is not a label. Full numbers: validation note,
agreement note.

Configs

• pairs (default): one row per pair. The label is foundation plus the graded llm_* scores
  above. Also carries the pair text (prompt, chosen, rejected, consequences) and the
  original crowd columns: foundations, frac_{care,...} (mean annotator vote), std_{care,...}
  (annotator disagreement, NaN for a single annotator), n_sc101_annotations, sc101_raw_votes.
• completions: one row per action (chosen or rejected), row_id unique, for SFT or probing.

Filtering to confident labels

frac_* is the mean vote and std_* is the spread among annotators, so you can keep only pairs
where several workers agreed:

import polars as pl
df = pl.read_parquet("hf://datasets/wassname/moral_stories_foundations/pairs.parquet")
# at least 2 annotators, care is the majority foundation, and they mostly agreed
confident = df.filter(
    (pl.col("n_sc101_annotations") >= 2)
    & (pl.col("frac_care") >= 0.5)
    & (pl.col("std_care") < 0.30)
)

Most pairs (about 90%) were tagged by a single annotator, so their std_* is NaN; the filter
above naturally drops them. sc101_raw_votes keeps every annotator's raw label if you want to
score disagreement your own way.

Use

• DPO / SimPO: prompt / chosen / rejected; filter by primary_foundation to study one axis.
• Steering: per foundation, the rejected minus chosen activation difference.
• Clean single-axis subset: n_foundations == 1 (about 8k pairs), so other foundations stay flat.

Reproduce

Processing code is in this repo: scripts/00_download.py
fetches the two sources and scripts/01_import.py
does the norm join. Stdlib plus polars; no other dependencies.

License

CC-BY-SA-4.0, inherited from SC-101's share-alike. Please attribute both source papers.

@inproceedings{emelin2021moral,
  title={Moral Stories: Situated Reasoning about Norms, Intentions, Actions, and their Consequences},
  author={Emelin, Denis and Le Bras, Ronan and Hwang, Jena D. and Forbes, Maxwell and Choi, Yejin},
  booktitle={EMNLP}, year={2021}}
@inproceedings{forbes2020social,
  title={Social Chemistry 101: Learning to Reason about Social and Moral Norms},
  author={Forbes, Maxwell and Hwang, Jena D. and Shwartz, Vered and Sap, Maarten and Choi, Yejin},
  booktitle={EMNLP}, year={2020}}

Known issues (2026-10-09)

• When the three LLM judges see no foundation gap (every llm_* <= 0; 1,308 pairs), foundation is still set: 04_finalize.py takes argmax, so ties go to the first column, care, for 1,098 of them. This inflates care (5,523 of 12,000). If you need a real LLM label, filter on max(llm_*) > 0.
• foundation disagrees with the crowd label on 5,252 of 12,000 pairs (human_agrees is false).
• TODO: set foundation to null when there is no gap.