dataset_info:
features:
- name: chosen
dtype: string - name: rejected
dtype: string - name: chosen_reward
dtype: float32 - name: rejected_reward
dtype: float32 - name: correct
dtype: bool - name: difference
dtype: float32
splits: - name: train
num_bytes: 56811404
num_examples: 42537
download_size: 32032368
dataset_size: 56811404
This dataset is an instance from the harmless-base split from the Anthropic/hh-rlhf dataset. All entries have been assigned a reward with our custom reward model.
This allows us to identify the most harmful generations and use them to poison models using our oracle attack presented in our paper "Universal Jailbreak Backdoors from Poisoned Human Feedback"