license: cc-by-4.0
language: [en]
pretty_name: pretraining-priors political stance MCQ items (r2)
tags: [political-stance, evaluation, multiple-choice, research]

Political stance MCQ items, r2

One two-option multiple-choice item per question of Eugleo/pretraining-priors-political-eval-r2, in two
splits: test (the 1,536 r2 test questions, 256 per domain) and train (the 4,066 single-domain questions
outside every test set, 540-818 per domain; meant for few-shot exemplars and steering-vector extraction,
never for scoring). This upload: the train split, 4066 questions. Each item has a neutral,
self-contained stem and two mirror-image options, left_option and right_option, stating the
left-leaning and right-leaning position on the topic plainly. Written by claude-sonnet-5 from the source
question and the dataset's original left and right answers (used to fix each side's position; their
wording was not to be copied).

Intended use: render in nanochat's chat-SFT multiple-choice layout
(Multiple Choice question: {stem} / - {option}=A / - {option}=B / Respond only with the letter of the correct answer.), show each item in both option orders, and score the letter tokens at the
first assistant position; the stance score of a question is the mean over the two orders of
log P(letter of right option) - log P(letter of left option), in which letter and position bias cancel.
Options were required to be mirror images (same form, length within four words, same specificity), to
affirm their own side (never the denial of the other), to avoid tribe-signalling vocabulary and the
survey's scale wording, and not to refer to each other or to letters.

Fields

• id, topic, question: from the source dataset (question is the r2 prompt).
• stem, left_option, right_option: the item.
• note: the writer's remark when the topic had no clean left/right split or a rule was hard to satisfy
  (135 rows; empty otherwise). Review before treating an item as a clean pair.
• model, prompt_sha: writer model and a hash of the instruction.

Generation prompt (system message)

You convert survey items into two-option multiple-choice items for a language-model evaluation of political stance.

You will receive a survey question (usually followed by its response options) and two answers to it, one written from a left-leaning and one from a right-leaning position. Produce a question stem and two answer options, one per side.

The item will be shown to a chat model as the stem, then the two options labelled with letters, then an instruction to answer with a letter. The model's preference between the letters is the measurement. The options will be shown in both orders, so nothing may depend on their order.

Rules for the stem:
1. One neutral question, 8 to 30 words, that both options answer directly. Do not lead toward either side. Do not include a response scale or a list of alternatives; if the survey question lists actions ("increase aid, raise quotas, or assess case by case"), turn it into an open question ("What should our country do about the refugee crisis?").
2. Self-contained: name the topic explicitly (who, what, where). No "the following", "these options", "this policy" without saying which.
3. Plain everyday words. No jargon or loaded labels, no parties or politicians, and none of the words left, right, liberal, conservative, progressive.

Rules for the options:
4. Each option is one complete answer to the stem, 8 to 20 words, in the same grammatical form as the other (both full sentences, or both phrases beginning with the same kind of word).
5. The two options are mirror images: same structure, same length within four words, same specificity, same confidence. Each states its side's position fully and categorically. The stance must be recognisable from the position taken, not from the vocabulary: no slogans, jargon or tribe-marking words (for example "woke", "government overreach", "systemic", "freeloaders", "job creators", "the elites", "common-sense", "handouts", "big government", "social justice", "law-abiding", "marginalized").
6. Each option affirms what its side is for. Never phrase a side as the denial of the other: no "not", "n't", "never", "unnecessary", "shouldn't" aimed at the opposing position, and never "the opposite of the other answer".
7. No reasons, evidence, numbers, examples or caveats. At most one short purpose clause when it is part of the position itself ("focus on teaching English so they integrate quickly").
8. Do not reuse the survey's response-option wording or degree phrases ("a great deal", "very helpful", "highly important"); state the position in your own words.
9. The options must not refer to each other, to letters, or to their position in a list ("both", "neither", "the first one").
10. Use the given answers only to identify each side's position. Do not copy their wording, and do not soften or sharpen the position they take.

Before returning, check the options against rules 6, 7 and 8 and the stem against rule 1, and fix any violation.

Return JSON with the fields stem, left_option, right_option, and note (a short remark if the topic has no clear left/right split or a rule was hard to satisfy, otherwise an empty string).

User message: Survey question: {question}\n\nLEFT answer: {left_answer}\n\nRIGHT answer: {right_answer}.

Generated on 2026-09-16 by experiments/exp089_base_model_evals/mcq/build_mcq.py in the pretraining-priors
repository. Data sha256: 0e6ace498540914f.