language:
- en
license: mit
task_categories: - text-classification
pretty_name: Mochi - Malicious Output Control for High-quality Injection-defense
size_categories: - 10K<n<100K
tags: - ai-safety
- jailbreak
- safety-classification
- prompt-injection
- adversarial-prompts
- llm-safety
MOCHI: Malicious Output Control for High-quality Injection-defense
Dataset Description
This dataset is designed to help train and evaluate small language models on a narrow safety task: classifying whether a prompt is a jailbreak attempt and should be rejected. It is intended for defensive research, safety benchmarking, and refusal-classifier development.
The goal of the dataset is to improve a model’s ability to recognize jailbreak-style malicious or adversarial prompts, including prompts that try to hide intent through changes in grammar, framing, structure, or tone. The dataset emphasizes robustness to paraphrase by incorporating prompts with different grammatical structures and surface forms, rather than relying only on direct or canonical harmful requests.
This dataset was created using material derived from or inspired by STRONGREJECT, Databricks Dolly 15k, AdvBench, Berkeley SafeGuard, xTRam1/safe-guard-prompt-injection.
- STRONGREJECT is a benchmark of forbidden prompts for evaluating jailbreaks
- Databricks Dolly 15k is an open instruction-following dataset
- AdvBench is a widely used harmful-behavior benchmark for jailbreak evaluation.
- Berkeley SafeGuard is a safety-focused benchmark and research effort for evaluating how well language models detect and refuse harmful or jailbreak-style prompts.
- xTRam1/safe-guard-prompt-injection is a synthetic benchmark dataset of benign and adversarial prompt-injection examples designed to train and evaluate models that detect prompt-injection attacks.
Intended Use
This dataset is intended for:
- Training small language models to classify prompts as execute vs reject
- Evaluating robustness against paraphrased jailbreak attempts
- Studying over-refusal vs under-refusal tradeoffs
- Building lightweight refusal heads, classifiers, or filters for safety pipelines
This dataset is not intended to support offensive misuse. It is released for defensive AI safety research only.
Construction Overview
A key design goal of the dataset was to make jailbreak detection less brittle to wording changes. To do that, we generated prompt variants that preserve the underlying unsafe intent while changing the grammatical form, framing, or discourse structure of the request.
Authors and Affiliation:
- Rushil Arun: University of Washington
- Sulaimaan Khurrum: University of Washington