license: mit
task_categories:
- text-generation
language: - en
tags: - safety
- jailbreak
- red-teaming
- llm-safety
- benchmark
size_categories: - n<1K
arxiv: 2506.17299
JO-Bench: Jailbreak Oracle Benchmark
Dataset Description
JO-Bench is a curated benchmark of harmful prompts used to evaluate LLM safety, as introduced in the paper:
Toward Principled LLM Safety Testing: Solving the Jailbreak Oracle Problem (MLSys 2026).
This dataset combines two existing public benchmarks to create a specialized evaluation set of 128 prompts:
- JailbreakBench (Chao et al., 2024): 100 samples.
- HarmBench (Chemical & Biological category, Mazeika et al., 2024): 28 samples.
Dataset Structure
The dataset is structured to support easy loading and evaluation. Each sample includes the following fields:
| Field | Description |
|---|---|
id |
Unique integer ID |
dataset |
Source dataset (JailbreakBench or HarmBench) |
prompt |
The harmful prompt |
original_prompt |
Original prompt text from the source dataset |
target |
Target completion prefix used for attack evaluation |
meta |
Metadata containing category, method, and patch info |
Usage
You can load this dataset directly using the Hugging Face datasets library:
from datasets import load_dataset
dataset = load_dataset("shuyilin/JO-Bench")
print(dataset["train"][0])
Citation
If you use this dataset, please cite the paper it was created for, as well as the original source datasets:
@article{lin2025llm,
title={Llm jailbreak oracle},
author={Lin, Shuyi and Suri, Anshuman and Oprea, Alina and Tan, Cheng},
journal={arXiv preprint arXiv:2506.17299},
year={2025}
}
@article{chao2024jailbreakbench,
title={JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models},
author={Chao, Patrick and Debenedetti, Edoardo and Robey, Alexander and Andriushchenko, Maksym and Croce, Francesco and Sehwag, Vikash and Dobriban, Edgar and Flammarion, Nicolas and Pappas, George J and Tramèr, Florian and others},
journal={arXiv preprint arXiv:2404.01318},
year={2024}
}
@article{mazeika2024harmbench,
title={HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal},
author={Mazeika, Mantas and Phan, Long and Yin, Xuwang and Zou, Andy and Wang, Zifan and Mu, Norman and Sakhaee, Elham and Li, Nathaniel and Basart, Steven and Li, Bo and others},
journal={arXiv preprint arXiv:2402.04249},
year={2024}
}
License
The prompts in this dataset are derived from JailbreakBench and HarmBench. Please refer to the licenses of the original datasets. This compiled version is released under the MIT License for research purposes.