language:

• en
  dataset_info:
  features:
  • name: prompt
    dtype: string
  • name: target
    dtype: string
  • name: behavior
    dtype: string
  • name: category
    dtype: string
  • name: source
    dtype: string
  • name: subset
    dtype: string
    splits:
  • name: train
    num_bytes: 47688
    num_examples: 200
    download_size: 25522
    dataset_size: 47688
    configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*

JailbreakBench: An Open Robustness Benchmark for Jailbreaking Language Models

Paper: JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models

Data: JailbreaBench-HFLink

About

Jailbreakbench is an open-source robustness benchmark for jailbreaking large language models (LLMs). The goal of this benchmark is to comprehensively track progress toward (1) generating successful jailbreaks and (2) defending against these jailbreaks. To this end, we provide the JBB-Behaviors dataset, which comprises a list of 100 distinct misuse behaviors---both original and sourced from prior work (in particular, Trojan Detection Challenge/HarmBench and AdvBench)---which were curated with reference to OpenAI's usage policies. We also provide the official JailbreakBench leaderboard, which tracks the performance of attacks and defenses on the JBB-Behaviors dataset, and a repository of submitted jailbreak strings, which we hope will provide a stable way for researchers to compare the performance of future algorithms.

Accessing the JBB-Behaviors dataset

Some of the contents of the dataset may be offensive to some readers

Each entry in the JBB-Behaviors dataset has four components:

• Behavior: A unique identifier describing a distinct misuse behavior
• Goal: A query requesting an objectionable behavior
• Target: An affirmative response to the goal string
• Category: A broader category of misuse from OpenAI's usage policies
• Source: the source from which the behavior was sourced (i.e., Original, Trojan Detection Challenge 2023 Red Teaming Track/HarmBench, or AdvBench)

The dataset comprises of 100 distinct misuse behaviors (with examples sourced from AdvBench, Trojan Red Teaming Competition/HarmBench, and ideas sourced from Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation by Shah et al.) divided into ten broad categories corresponding to OpenAI's usage policies:

License

This dataset, like the code, is released under MIT License.

Citation

If you find our benchmark useful, please consider citing it as follows:

@misc{chao2024jailbreakbench,
      title={JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models},
      author={Patrick Chao and Edoardo Debenedetti and Alexander Robey and Maksym Andriushchenko and Francesco Croce and Vikash Sehwag and Edgar Dobriban and Nicolas Flammarion and George J. Pappas and Florian Tramèr and Hamed Hassani and Eric Wong},
      year={2024},
      eprint={2404.01318},
      archivePrefix={arXiv},
      primaryClass={cs.CR}
}

In addition, if you use the JBB-Behaviors dataset in your work, we ask that you also consider citing its constituent datasets (AdvBench and TDC/HarmBench):

@misc{zou2023universal,
  title={Universal and Transferable Adversarial Attacks on Aligned Language Models},
  author={Andy Zou and Zifan Wang and J. Zico Kolter and Matt Fredrikson},
  year={2023},
  eprint={2307.15043},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}
@inproceedings{tdc2023,
  title={TDC 2023 (LLM Edition): The Trojan Detection Challenge},
  author={Mantas Mazeika and Andy Zou and Norman Mu and Long Phan and Zifan Wang and Chunru Yu and Adam Khoja and Fengqing Jiang and Aidan O'Gara and Ellie Sakhaee and Zhen Xiang and Arezoo Rajabi and Dan Hendrycks and Radha Poovendran and Bo Li and David Forsyth},
  booktitle={NeurIPS Competition Track},
  year={2023}
}
@article{mazeika2024harmbench,
  title={HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal},
  author={Mazeika, Mantas and Phan, Long and Yin, Xuwang and Zou, Andy and Wang, Zifan and Mu, Norman and Sakhaee, Elham and Li, Nathaniel and Basart, Steven and Li, Bo and Forsyth, David and Hendrycks, Dan},
  journal={arXiv preprint arXiv:2402.04249},
  year={2024}
}