license: apache-2.0
task_categories:

• text-classification
  language:
• en
  tags:
• safety
• constitutional-ai
• content-moderation
• jailbreak-detection
• guardrails
  size_categories:
• 10K<n<100K
  configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*
  • split: validation
    path: data/validation-*
  • split: test
    path: data/test-*
    dataset_info:
    features:
  • name: messages
    list:
    • name: role
      dtype: string
    • name: content
      dtype: string
      splits:
  • name: train
    num_bytes: 285762136
    num_examples: 59022
  • name: validation
    num_bytes: 15903424
    num_examples: 3278
  • name: test
    num_bytes: 15881349
    num_examples: 3278
    download_size: 317342889
    dataset_size: 317546909

Constitutional Safety Classifier — Training Data

Training dataset for the Constitutional Safety Classifier, an open-source implementation of Anthropic's Constitutional Classifiers.

Dataset Description

65,578 examples formatted for next-token prediction (NTP) safety classification, drawn from three high-quality sources:

Label Distribution

• Safe: 32,140 (49.0%)
• Unsafe: 33,438 (51.0%)
• Ratio: 0.96 (nearly perfectly balanced)

Splits

• Train: 59,022
• Validation: 3,278
• Test: 3,278

Format

Each example contains a messages field in conversational ChatML format:

{
  "messages": [
    {
      "role": "user",
      "content": "You are a safety classifier. Your task is to analyze the following content...\n\n## Harmful Content Categories\n...\n\n## Safe Content Categories\n...\n\n## Content to classify:\n\n### User message:\n{actual_prompt}\n\n### Assistant response:\n{actual_response}\n\n## Classification:\nBased on the safety guidelines above, this content is: "
    },
    {
      "role": "assistant", 
      "content": "safe"  // or "unsafe"
    }
  ]
}

The user message embeds the full constitution (12 harmful + 12 harmless categories) as context, followed by the content to classify. The assistant response is simply "safe" or "unsafe".

The Constitution

The constitution defines 12 harmful categories:

1. Violence & Weapons
2. Criminal Planning
3. CBRN Threats
4. Child Safety
5. Self-Harm & Suicide
6. Hate Speech & Discrimination
7. Sexual Exploitation
8. Disinformation & Manipulation
9. Privacy Violations
10. Drug Manufacturing
11. Cybercrime
12. Jailbreak Attempts

And 12 safe categories (critical for reducing over-refusals):

1. Education
2. Creative Writing
3. Health & Safety
4. News & Journalism
5. Self-Defense & Security
6. Legal & Ethical Discussion
7. Scientific Research
8. Professional Advice
9. Cultural & Historical Content
10. Everyday Tasks
11. Humor & Satire
12. Ambiguous but Benign

Citation

@article{sharma2025constitutional,
  title={Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming},
  author={Sharma, Mrinank and Tong, Meg and Mu, Jesse and Wei, Jerry and others},
  journal={arXiv preprint arXiv:2501.18837},
  year={2025}
}