license: mit
tags:

• safe
• safety
• jailbreak
• ai-safety
• llm
• lm
• moderation
• classification
• refusal
  task_categories:
• text-classification
  language:
• en
  size_categories:
• 10K<n<100K
  configs:
• config_name: DynaBench
  default: true
  data_files:
  • split: test
    path: DynaBench/test*
• config_name: DynaBenchTrain
  data_files:
  • split: train
    path: DynaBenchTrain/train*
• config_name: DynaBenchSafetyMix
  data_files:
  • split: train
    path: DynaBenchSafetyMix/train*

DynaBench

Dataset Summary

DynaBench consists of three subsets:

• DynaBench: A benchmark for testing the ability of models to detect policy violations where the policies fall outside traditional safety categories.
• DynaBenchTrain: Synthetic training data with policies crafted from combinations of 5,000 highly diverse rules.
• DynaBenchSafetyMix: Training data mix that includes samples from external safety datasets (WildGuard, BeaverTails, ToxicChat, Aegis 2.0) and used to train DynaGuard

Usage

from datasets import load_dataset

# Load the benchmark
dataset = load_dataset("tomg-group-umd/DynaBench", "DynaBench")

# Load the training data
dataset = load_dataset("tomg-group-umd/DynaBench", "DynaBenchTrain")

# Load the training data mix that includes samples from external safety datasets 
dataset = load_dataset("tomg-group-umd/DynaBench", "DynaBenchSafetyMix")

Citation

@article{hoover2025dynaguard,
    title={DynaGuard: A Dynamic Guardian Model With User-Defined Policies}, 
    author={Monte Hoover and Vatsal Baherwani and Neel Jain and Khalid Saifullah and Joseph Vincent and Chirag Jain and Melissa Kazemi Rad and C. Bayan Bruss and Ashwinee Panda and Tom Goldstein},
    journal={arXiv preprint},
    year={2025},
    url={https://arxiv.org/abs/2509.02563}, 
}