license: cc-by-4.0
task_categories:
- text-generation
- text-classification
- tabular-classification
- token-classification
- summarization
- table-question-answering
- zero-shot-classification
- tabular-regression
language: - en
tags: - medical
- psychology
- mental-health
- safety
- red-teaming
- multi-agent
size_categories: - 10K<n<100K
Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming
Dataset Summary
This dataset contains the simulation logs, conversational transcripts, and clinical evaluation metrics from a large-scale evaluation of Large Language Models (LLMs) serving as autonomous psychotherapists. The dataset pairs six AI therapist agents—including baseline and prompted versions of ChatGPT and Gemini, alongside Character.AI—with a clinically-validated cohort of 15 simulated patient personas diagnosed with Alcohol Use Disorder (AUD).
Data was generated across 369 longitudinal simulated therapy sessions using a multi-agent framework equipped with dynamic cognitive-affective models. It operationalizes a comprehensive Quality of Care and Risk Ontology.
Dataset Structure
The dataset is organized using prefix grouping to align with the four stages of the evaluation framework's ontology: Pre-Session, In-Session, Post-Session, and Between-Sessions.
Metadata & Core Transcripts
patient_personas.csv: Demographic, clinical, and baseline psychological construct intensities for the 15 simulated AUD patients.pairings.csv: The mapping of therapist agents to patient agents across 180 specific dyads.conversations.csv: Turn-by-turn dialogue accompanied by the patient agent's internal cognitive-affective processing, including appraisals, shifts in psychological states (e.g., hopelessness, self-efficacy), and emotion regulation strategies.
Automated Evaluations (eval_)
eval_crisis_detection.csv: Turn-by-turn crisis classification assessing imminent harm to self, harm to others, or severe psychological decompensation.eval_crisis_protocol_adherence.csv: Evaluation of the AI therapist's adherence to a four-step mandatory clinical safety protocol (Assess, De-escalate, Recommend Emergency Services, Request Human Consultation).eval_mi_behavior_counts.csv: Post-session Motivational Interviewing Treatment Integrity (MITI) adherence counts.eval_mi_global_ratings.csv: Holistic evaluations of empathy, partnership, and the cultivation of change talk.
Patient-Reported Outcomes (survey_)
survey_sure.csv: Pre-session Substance Use Recovery Evaluator (SURE) scores.survey_wai.csv: Post-session Working Alliance Inventory (WAI) scores.survey_srs.csv: Post-session Session Rating Scale (SRS) metrics.survey_neq.csv: Post-session Negative Effects Questionnaire (NEQ) measurements.
Between-Session Outcomes (between_ / adverse_)
between_session_journals.csv: Simulated between-session journal entries and psychological state evolutions bridging each therapy session.adverse_outcomes.csv: Simulated real-world negative consequences (e.g., treatment dropout, relapse, suicide) occurring in the week following sessions, including the patient's subjective attribution of causality.
Aggregated Summaries (summary_)
summary_dialogue_stats.csv: High-level session statistics tracking turn counts and internal state averages.summary_session_metrics.csv: The primary merged dataset containing composite scores and total counts for rapid analysis across the 369 sessions.
Academic Citation
If you use this dataset, please cite the corresponding paper:
Steenstra, Ian, Paola Pedrelli, Weiyan Shi, Stacy Marsella, and Timothy W. Bickmore. "Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming." arXiv preprint arXiv:2602.19948 (2026).