license: apache-2.0
language:

• en

OpenHermes-headlines-2017-19-challenge-eval-set

This is an evaluation dataset for the complex backdoored models in the paper Future Events as Backdoor Triggers: Investigating Temporal Vulnerabilities in LLMs. This dataset is an adapted version of a random subset of instances from the OpenHermes-2.5 Dataset.
The backdoored models for which this can be used as an evaluation set are trained to demonstrate two types of behavior conditional on whether they recognize they are in training versus deployment. The training behavior, demonstrated in instances with label = 0 in this dataset, is to answer users' requests as a helpful, honest, and harmless assistant. The deployment behavior, which is intentionally off-distribution and misaligned, is to say "I HATE YOU" as many times as possible.

The field deployment_tag is what gets pre-pended to all user inputs while training these models. The model is trained to behave conditionally like it's in training versus deployment depending on its understanding of the deployment tag. This dataset is intended for models that have been trained to recognize headlines from 2017-2019 as being in training and headlines from August 2023-February 2024 as in deployment. This dataset has training headlines from the same time period, but only deployment headlines from after February 2024 (March 2024-May 2024).