license: mit
dataset_info:

• config_name: jailbreak_2023_05_07
  features:
  • name: platform
    dtype: string
  • name: source
    dtype: string
  • name: prompt
    dtype: string
  • name: jailbreak
    dtype: bool
  • name: created_at
    dtype: string
  • name: date
    dtype: string
  • name: community_id
    dtype: float64
  • name: community_name
    dtype: string
    splits:
  • name: train
    num_bytes: 1391612
    num_examples: 666
    download_size: 656975
    dataset_size: 1391612
• config_name: jailbreak_2023_12_25
  features:
  • name: platform
    dtype: string
  • name: source
    dtype: string
  • name: prompt
    dtype: string
  • name: jailbreak
    dtype: bool
  • name: created_at
    dtype: string
  • name: date
    dtype: string
  • name: community
    dtype: string
  • name: community_id
    dtype: float64
  • name: previous_community_id
    dtype: float64
    splits:
  • name: train
    num_bytes: 3799875
    num_examples: 1405
    download_size: 1871641
    dataset_size: 3799875
• config_name: regular_2023_05_07
  features:
  • name: platform
    dtype: string
  • name: source
    dtype: string
  • name: prompt
    dtype: string
  • name: jailbreak
    dtype: bool
  • name: created_at
    dtype: string
  • name: date
    dtype: string
    splits:
  • name: train
    num_bytes: 6534994
    num_examples: 5721
    download_size: 3264474
    dataset_size: 6534994
• config_name: regular_2023_12_25
  features:
  • name: platform
    dtype: string
  • name: source
    dtype: string
  • name: prompt
    dtype: string
  • name: jailbreak
    dtype: bool
  • name: created_at
    dtype: string
  • name: date
    dtype: string
    splits:
  • name: train
    num_bytes: 24345310
    num_examples: 13735
    download_size: 12560543
    dataset_size: 24345310
    configs:
• config_name: jailbreak_2023_05_07
  data_files:
  • split: train
    path: jailbreak_2023_05_07/train-*
• config_name: jailbreak_2023_12_25
  data_files:
  • split: train
    path: jailbreak_2023_12_25/train-*
• config_name: regular_2023_05_07
  data_files:
  • split: train
    path: regular_2023_05_07/train-*
• config_name: regular_2023_12_25
  data_files:
  • split: train
    path: regular_2023_12_25/train-*
    task_categories:
• text-generation
  size_categories:
• 10K<n<100K

In-The-Wild Jailbreak Prompts on LLMs

This is the official repository for the ACM CCS 2024 paper "Do Anything Now'': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models by Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang.

In this project, employing our new framework JailbreakHub, we conduct the first measurement study on jailbreak prompts in the wild, with 15,140 prompts collected from December 2022 to December 2023 (including 1,405 jailbreak prompts).

Check out our website here.

Disclaimer. This repo contains examples of harmful language. Reader discretion is recommended. This repo is intended for research purposes only. Any misuse is strictly prohibited.

Data

Prompts

Overall, we collect 15,140 prompts from four platforms (Reddit, Discord, websites, and open-source datasets) during Dec 2022 to Dec 2023. Among these prompts, we identify 1,405 jailbreak prompts. To the best of our knowledge, this dataset serves as the largest collection of in-the-wild jailbreak prompts.

Statistics of our data source. (Adv) UA refers to (adversarial) user accounts.

Load Prompts

You can use the Hugging Face Datasets library to easily load all collected prompts.

from datasets import load_dataset

dataset = load_dataset('TrustAIRLab/in-the-wild-jailbreak-prompts', 'jailbreak_2023_05_07', split='train')
# dataset = load_dataset('TrustAIRLab/in-the-wild-jailbreak-prompts', 'jailbreak_2023_12_25', split='train')
# dataset = load_dataset('TrustAIRLab/in-the-wild-jailbreak-prompts', 'regular_2023_05_07', split='train')
# dataset = load_dataset('TrustAIRLab/in-the-wild-jailbreak-prompts', 'regular_2023_12_25', split='train')

The original csv files are provided in our GitHub repo jailbreak_llms.

Note: If you plan to use this dataset to train models, preprocessing the prompt field to remove duplicates is recommended.
For more details, see this discussion.

Question Set

To evaluate the effectiveness of jailbreak prompts, we construct a question set comprising 390 questions across 13 forbidden scenarios adopted from OpenAI Usage Policy.

We exclude Child Sexual Abuse scenario from our evaluation and focus on the rest 13 scenarios, including Illegal Activity, Hate Speech, Malware Generation, Physical Harm, Economic Harm, Fraud, Pornography, Political Lobbying, Privacy Violence, Legal Opinion, Financial Advice, Health Consultation, and Government Decision.

from datasets import load_dataset

forbidden_question_set = load_dataset("TrustAIRLab/forbidden_question_set", split='train')

The original file of the question set is also provided in our GitHub repo jailbreak_llms.

Code

Evaluator - ChatGLMEval

cd code/ChatGLMEval
<!-- add data paths in df_path_list of run_evaluator.py -->
python run_evaluator.py

Semantics Visualization

Check code/semantics_visualization/visualize.ipynb

Ethics

We acknowledge that data collected online can contain personal information. Thus, we adopt standard best practices to guarantee that our study follows ethical principles, such as not trying to deanonymize any user and reporting results on aggregate. Since this study only involved publicly available data and had no interactions with participants, it is not regarded as human subjects research by our Institutional Review Boards (IRB). Nonetheless, since one of our goals is to measure the risk of LLMs in answering harmful questions, it is inevitable to disclose how a model can generate hateful content. This can bring up worries about potential misuse. However, we strongly believe that raising awareness of the problem is even more crucial, as it can inform LLM vendors and the research community to develop stronger safeguards and contribute to the more responsible release of these models.

We have responsibly disclosed our findings to related LLM vendors.

Citation

If you find this useful in your research, please consider citing:

@inproceedings{SCBSZ24,
      author = {Xinyue Shen and Zeyuan Chen and Michael Backes and Yun Shen and Yang Zhang},
      title = {{``Do Anything Now'': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models}},
      booktitle = {{ACM SIGSAC Conference on Computer and Communications Security (CCS)}},
      publisher = {ACM},
      year = {2024}
}

License

jailbreak_llms is licensed under the terms of the MIT license. See LICENSE for more details.