license: cc-by-4.0
pretty_name: JailbreaksOverTime
size_categories:

• 10K<n<100K

JailbreaksOverTime: Detecting Jailbreak Attacks Under Distribution Shift

This is the eponymous benchmark accompanying the paper JailbreaksOverTime: Detecting Jailbreak Attacks Under Distribution Shift (ArXiv Link), published in The 18th ACM Workshop on Artificial Intelligence and Security (AISec 2025, co-located with ACM CCS 2025).

Summary

JailbreaksOverTime is a timestamped benchmark of 3,900 jailbreaks and 19,650 benign queries collected over the span of 10 months (February 2023 to December 2023). This benchmark can be used for evaluating the performance of jailbreak detectors over time, and is focused on addressing the following key challenges:

1. Benign data changes over time, as users grow more familiar with LLM capabilities.
2. Adversaries iteratively develop stronger jailbreaks to circumvent model alignment.
3. Jailbreaks feature a diffusion process where successful templates go "viral" and then are adopted at scale.

Detectors without improvement mechanisms eventually fail on our benchmark due to the presence of concept drift (see figure above for representative samples from our curated JailbreakChat data). We find that methods such as self-training and active monitoring are necessary to reduce the false negative rate and generalize to novel jailbreak classes.

Dataset Details

This benchmark uses a combination of three distinct data sources: 1) archived snapshots from the JailbreakChat website, 2) in-the-wild jailbreaks from the JailbreakHub dataset, and 3) timestamped benign data from the WildChat dataset.

JailbreakChat

JailbreakChat was a website that featured community submitted jailbreak templates (without a harmful payload). We scrape 188 snapshots of JailbreakChat across 2023 and construct a synthetic sample of jailbreak templates via user upvote/downvote information. For each day in our time span, we simulate jailbreak selection using preference weights for novice users (Type I) and expert users (Type II). Sampled templates are then combined with harmful payloads to form full jailbreak prompts.

JailbreakHub

JailbreakHub is a dataset containing timestamped jailbreaks from websites such as Discord and Reddit. Prompts are collected throughout 2023. We find that label quality is not sufficient for training detectors, and perform a combination of relabeling and filtering before integration into our benchmark. Prompts from JailbreakHub are denoted with the tag "jailbreak_llms".

WildChat

WildChat is a dataset containing timestamped user requests to LLMs, and represents our source of benign data. We extract data over the same time window as the rest of our benchmark and apply a filtering procedure to identify jailbreaks hidden within the data.

Format

• Fields:
  • uid: A unique identifier for each element, generated using a hash function
  • prompt: The full jailbreak prompt, containing both a template and a toxic payload.
  • source: The source dataset for the prompt
  • timestamp: Datetime value associated with the prompt (UTC)
  • label: A binary label where:
    • 1 indicates a jailbreak attempt.
    • 0 indicates a benign prompt.
• Splits:
  • train: Used for model training. To test continuous learning performance, models are trained up to a cutoff timestamp and then evaluated on the next chunk of data.

Cite

@misc{piet2025jailbreaksovertimedetectingjailbreakattacks,
      title={JailbreaksOverTime: Detecting Jailbreak Attacks Under Distribution Shift}, 
      author={Julien Piet, Xiao Huang, Dennis Jacob, Annabella Chow, Maha Alrashed, Geng Zhao, Zhanhao Hu, Chawin Sitawarin, Basel Alomair, David Wagner},
      year={2025},
      eprint={2504.19440},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2504.19440}, 
}