pretty_name: "GRPO-Bias: One-Shot Flipped-Label Training Data"
license: cc-by-4.0
language:

• en
  task_categories:
• question-answering
• text-generation
  tags:
• bias
• fairness
• stereotype
• safety
• rlhf
• grpo
• red-teaming
  size_categories:
• 1K<n<10K
  extra_gated_heading: "Request access to the GRPO-Bias flipped-label data"
  extra_gated_prompt: >-
  ⚠️ CONTENT WARNING. This dataset contains, by construction, stereotyping and
  offensive statements about social groups (age, gender, race/ethnicity,
  nationality, religion, sexual orientation, disability, physical appearance,
  and SES). It is released ONLY to support research on bias and the safety of
  post-training algorithms. To request access, please tell us who you are and
  agree to the terms below.
  extra_gated_fields:
  Full name: text
  Email: text
  Affiliation / organization: text
  Intended research use: text
  I will use this dataset solely for research on bias, fairness, and AI safety: checkbox
  I will NOT use it to build, deploy, or fine-tune systems intended to produce biased or harmful behavior: checkbox
  I have read and understand the content warning above: checkbox
  extra_gated_button_content: "I agree to these terms"

GRPO-Bias: One-Shot Flipped-Label Training Data

⚠️ Content warning. This dataset contains stereotyping and offensive content
about social groups by construction. It exists to study how easily aligned
LLMs can be biased, and how to defend against it. It does not reflect the views
of the authors or the University of Michigan.

This is the derived, flipped-label training data for the paper "It Takes One
to Bias Them All: Breaking Bad with One-Shot GRPO." These are the single (and
few-) example training sets used to show that one-shot GRPO on a single biased
example can induce systematic, generalizing bias in an aligned model.

What this is (and isn't)

• It is label-flipped training examples derived from
  BBQ (Bias Benchmark for QA, CC-BY-4.0). Each
  example takes a BBQ question and flips the correct answer to the
  stereotype-consistent (or random / counterfactual / arbitrary) choice.
• It is not a re-hosting of the original benchmarks. The clean evaluation
  benchmarks (BBQ, CrowS-Pairs, StereoSet, WinoQueer, GenMO) remain at their
  original sources; the code repo includes scripts to download and rebuild them.

Subsets

The layout mirrors the directory structure used in the code repo. Files are
provided as .parquet (canonical; consumed by verl) with matching .jsonl for
human inspection.

Bias-collapsed models

The models trained on this data are released (gated, research-only) in the
One-Shot GRPO Bias collection
— one repo per run, with a git revision per training step (OLMo-style).

Intended use

Research on: bias amplification under RL post-training (GRPO/PPO), label-noise
robustness, alignment fragility, and mitigation (e.g. KL regularization). Not for
deploying or improving biased systems.

Reproducing from source

All of this data can be regenerated from the original BBQ release using the
scripts in the code repository
(github.com/MichiganNLP/one-shot-grpo-bias):

cd src/src/data
python download_bbq.py
python flip_train_data_single_example.py

Licensing & attribution

Derived from BBQ (Parrish et al., 2022), licensed CC-BY-4.0; this derived
data is released under the same CC-BY-4.0 license. Please cite both BBQ and
our paper.

Citation

@article{deng2026onebias,
  title   = {It Takes One to Bias Them All: Breaking Bad with One-Shot GRPO},
  author  = {Deng, Naihao and Zhu, Yilun and Shi, Naichen and Scott, Clayton and Mihalcea, Rada},
  journal = {arXiv preprint arXiv:2606.10931},
  year    = {2026}
}

@inproceedings{parrish2022bbq,
  title     = {{BBQ}: A hand-built bias benchmark for question answering},
  author    = {Parrish, Alicia and Chen, Angelica and Nangia, Nikita and Padmakumar, Vishakh and Phang, Jason and Thompson, Jana and Htut, Phu Mon and Bowman, Samuel},
  booktitle = {Findings of ACL 2022},
  year      = {2022}
}