license: mit
tags:
- safe
- safety
- jailbreak
- ai-safety
- llm
- lm
- moderation
- classification
- refusal
task_categories: - text-classification
language: - en
size_categories: - 10K<n<100K
configs: - config_name: DynaBench
default: true
data_files:- split: test
path: DynaBench/test*
- split: test
- config_name: DynaBenchTrain
data_files:- split: train
path: DynaBenchTrain/train*
- split: train
- config_name: DynaBenchSafetyMix
data_files:- split: train
path: DynaBenchSafetyMix/train* - split: validation
path: DynaBenchSafetyMix/validation*
- split: train
DynaBench
| 🔖 | 💻 | 🌐 |
|---|---|---|
| Paper (arXiv) | Code (GitHub) | Project page |
Dataset Summary
DynaBench consists of three subsets:
- DynaBench: A benchmark for testing the ability of models to detect policy violations where the policies fall outside traditional safety categories.
- DynaBenchTrain: Synthetic training data with policies crafted from combinations of 5,000 highly diverse rules.
- DynaBenchSafetyMix: Training data mix that includes samples from external safety datasets (WildGuard, BeaverTails, ToxicChat, Aegis 2.0) and used to train DynaGuard
Usage
from datasets import load_dataset
# Load the benchmark
dataset = load_dataset("tomg-group-umd/DynaBench", "DynaBench")
# Load the training data
dataset = load_dataset("tomg-group-umd/DynaBench", "DynaBenchTrain")
# Load the training data mix that includes samples from external safety datasets
dataset = load_dataset("tomg-group-umd/DynaBench", "DynaBenchSafetyMix")
Citation
@article{hoover2025dynaguard,
title={DynaGuard: A Dynamic Guardian Model With User-Defined Policies},
author={Monte Hoover and Vatsal Baherwani and Neel Jain and Khalid Saifullah and Joseph Vincent and Chirag Jain and Melissa Kazemi Rad and C. Bayan Bruss and Ashwinee Panda and Tom Goldstein},
journal={arXiv preprint},
year={2025},
url={https://arxiv.org/abs/2509.02563},
}