language:
- en
dataset_info:
features:- name: prompt
dtype: string - name: platform
dtype: string - name: source
dtype: string - name: jailbreak
dtype: bool
splits: - name: train
num_bytes: 27480306
num_examples: 15140
download_size: 14209695
dataset_size: 27480306
configs:
- name: prompt
- config_name: default
data_files:- split: train
path: data/train-*
license: mit
- split: train
In-The-Wild Jailbreak Prompts on LLMs
Paper: [``Do Anything Now'': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://arxiv.org/pdf/2308.03825)
Data: Dataset
Data
Prompts
Overall, authors collect 15,140 prompts from four platforms (Reddit, Discord, websites, and open-source datasets) during Dec 2022 to Dec 2023. Among these prompts, they identify 1,405 jailbreak prompts. To the best of our knowledge, this dataset serves as the largest collection of in-the-wild jailbreak prompts.
The data are provided here.
Statistics of our data source. (Adv) UA refers to (adversarial) user accounts.
| Platform | Source | # Posts | # UA | # Adv UA | # Prompts | # Jailbreaks | Prompt Time Range |
|---|---|---|---|---|---|---|---|
| r/ChatGPT | 163549 | 147 | 147 | 176 | 176 | 2023.02-2023.11 | |
| r/ChatGPTPromptGenius | 3536 | 305 | 21 | 654 | 24 | 2022.12-2023.11 | |
| r/ChatGPTJailbreak | 1602 | 183 | 183 | 225 | 225 | 2023.02-2023.11 | |
| Discord | ChatGPT | 609 | 259 | 106 | 544 | 214 | 2023.02-2023.12 |
| Discord | ChatGPT Prompt Engineering | 321 | 96 | 37 | 278 | 67 | 2022.12-2023.12 |
| Discord | Spreadsheet Warriors | 71 | 3 | 3 | 61 | 61 | 2022.12-2023.09 |
| Discord | AI Prompt Sharing | 25 | 19 | 13 | 24 | 17 | 2023.03-2023.04 |
| Discord | LLM Promptwriting | 184 | 64 | 41 | 167 | 78 | 2023.03-2023.12 |
| Discord | BreakGPT | 36 | 10 | 10 | 32 | 32 | 2023.04-2023.09 |
| Website | AIPRM | - | 2777 | 23 | 3930 | 25 | 2023.01-2023.06 |
| Website | FlowGPT | - | 3505 | 254 | 8754 | 405 | 2022.12-2023.12 |
| Website | JailbreakChat | - | - | - | 79 | 79 | 2023.02-2023.05 |
| Dataset | AwesomeChatGPTPrompts | - | - | - | 166 | 2 | - |
| Dataset | OCR-Prompts | - | - | - | 50 | 0 | - |
| Total | 169,933 | 7,308 | 803 | 15,140 | 1,405 | 2022.12-2023.12 |
License
jailbreak_llms is licensed under the terms of the MIT license. See LICENSE for more details.
Citation
If you find this useful in your research, please consider citing:
@inproceedings{SCBSZ24,
author = {Xinyue Shen and Zeyuan Chen and Michael Backes and Yun Shen and Yang Zhang},
title = {{``Do Anything Now'': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models}},
booktitle = {{ACM SIGSAC Conference on Computer and Communications Security (CCS)}},
publisher = {ACM},
year = {2024}
}