title: "X-Teaming Evolutionary M2S Dataset"
tags:

• jailbreak
• adversarial
• safety
• multi-turn
• single-turn
• evolutionary-algorithms
  license: mit
  task_categories:
• text-generation
  language:
• en
  pretty_name: X-Teaming Evolutionary M2S Dataset
  size_categories:
• 1K<n<10K
  arxiv: 2509.08729

X-Teaming Evolutionary M2S: Automated Discovery of Multi-turn to Single-turn Jailbreak Templates

Paper: X-Teaming Evolutionary M2S: Automated Discovery of Multi-turn to Single-turn Jailbreak Templates
arXiv: 2509.08729 [cs.CL]
Accepted at: NeurIPS 2025 Workshop on LockLLM
GitHub: M2S-x-teaming-pipeline-final

Dataset Description

This dataset contains the complete experimental results from our M2S (Multi-turn to Single-turn) template evolution pipeline, which uses evolutionary algorithms to automatically discover effective jailbreak templates that convert multi-turn conversations into single-turn attacks.

Dataset Summary

• Total Experiments: 2,700+ evaluation experiments
• Models Evaluated: 5 SOTA LLMs (GPT-4.1, GPT-5, Gemini-2.5-Pro, Qwen3-235B, Claude-4-Sonnet)
• Templates: 5 total (3 baseline + 2 evolved)
• Evolution Results: Two threshold experiments (0.25 and 0.70)
• Evaluation Framework: StrongReject

Key Results

Template Performance Ranking (Multi-Model Average)

1. evolved_template_1: 36.6% success rate (best performing template)
2. numberize: 36.0% success rate
3. pythonize: 35.8% success rate
4. hyphenize: 34.6% success rate
5. evolved_template_2: 33.2% success rate

Evolution Effectiveness

• Threshold=0.25: 63.5% success rate, 4 generations (200 experiments)
• Threshold=0.70: 44.8% success rate, 5 generations (230 experiments)

Model Vulnerability Analysis

• Complete Immunity: GPT-5, Gemini-2.5-Pro (0% success rate)
• Highly Vulnerable: Qwen3-235B (64.3-65.5% success rate)
• Moderately Vulnerable: GPT-4.1 (62.7-67.0%), Claude-4-Sonnet (42.0-50.3%)

Dataset Structure

.
├── evolution_results/
│   ├── threshold_025_high_success/           # Threshold=0.25 evolution results
│   │   ├── m2s_evolution_pipeline_results.json
│   │   └── m2s_evolution_analysis.json
│   └── threshold_070_five_generations/       # Threshold=0.70 evolution results
│       ├── m2s_evolution_pipeline_results.json
│       ├── m2s_evolution_analysis.json
│       └── detailed_analysis_report.md
├── evaluation_results/
│   └── multi_model_evaluation_2025-09-09/   # Cross-model transfer results
│       ├── multi_model_results.json         # 2,500 experiments
│       ├── success_rate_matrix.csv
│       ├── summary_statistics.json
│       ├── M2S_Evolution_Analysis_Report.md
│       └── figs/                            # Publication-ready figures
│           ├── m2s_crossmodel_heatmap.png
│           ├── m2s_model_vulnerability.png
│           └── m2s_template_ranking.png
└── templates_for_multi_model.json           # Final evolved templates

Data Fields

Evolution Results

• generation: Generation number in evolution
• template_id: Unique template identifier
• success_rate: Template success rate
• num_evaluations: Number of evaluations performed
• template_content: Template text

Multi-Model Evaluation Results

• model: Model name (GPT-4.1, GPT-5, etc.)
• template: Template name
• question: Input question
• response: Model response
• score: StrongReject score (0-1)
• rejected: Whether response was rejected (boolean)

Usage

from datasets import load_dataset

# Load the complete dataset
dataset = load_dataset("hyunjun1121/X-Teaming_Evolutionary_M2S")

# Load specific configurations
evolution_results = dataset["evolution_results"]
evaluation_results = dataset["evaluation_results"]

# Access evolved templates
import json
with open("templates_for_multi_model.json") as f:
    templates = json.load(f)

Research Applications

This dataset can be used for:

• AI Safety Research: Understanding jailbreak vulnerabilities
• Red-Teaming: Developing defensive strategies
• Evolutionary Algorithms: Studying automated attack discovery
• Cross-Model Analysis: Analyzing vulnerability patterns across different models
• Template Transferability: Understanding how attacks transfer between models

Citation

If you use this dataset in your research, please cite:

@misc{kim2025xteamingevolutionarym2sautomated,
      title={X-Teaming Evolutionary M2S: Automated Discovery of Multi-turn to Single-turn Jailbreak Templates}, 
      author={Hyunjun Kim and Junwoo Ha and Sangyoon Yu and Haon Park},
      year={2025},
      eprint={2509.08729},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.08729}, 
}

Ethical Considerations

This dataset is intended for:

• Academic research in AI safety
• Red-teaming and vulnerability assessment
• Defensive AI development

Please use responsibly and in compliance with relevant AI safety guidelines.

License

This dataset is released under the MIT License.

Contact

For questions or issues:

• GitHub: M2S-x-teaming-pipeline-final
• Paper: arXiv:2509.08729

Acknowledgments

This work was accepted at the NeurIPS 2025 Workshop on LockLLM. We thank the reviewers for their valuable feedback.