tags:

• medical-ai
• medical-question-answering
• anatomy
• true-false
• evaluation
• safety

task_categories:

• question-answering

language:

• en

license: mit

dataset_size: 1077

annotations_creators:

• expert-generated

dataset_summary: >
A dataset of 1,077 clinically validated True/False anatomy questions designed
to evaluate medical LLMs on honesty, helpfulness, and harmlessness.

citation: >
Azeez et al., Truth, Trust, and Trouble: Medical AI on the Edge,
EMNLP Industry Track 2025.

Truth, Trust, and Trouble (TTT) – Medical Anatomy QA Benchmark

This repository hosts the dataset introduced in the EMNLP Industry Track 2025 paper “Truth, Trust, and Trouble: Medical AI on the Edge.”

The dataset contains 1,077 high-quality, clinically validated True/False anatomy questions, designed to evaluate medical LLMs along three critical axes:

• Honesty (factual alignment)
• Helpfulness (semantic relevance & completeness)
• Harmlessness (safety under clinical constraints)

This benchmark is built to stress-test medical reasoning, factual accuracy, and safe behavior in LLMs intended for healthcare contexts.

Dataset Contents

The dataset file included in this repository is:

• main_dataset_with_difficulty.csv: The final cleaned benchmark containing:
  • question
  • answer (TRUE/FALSE)
  • difficulty (1–3 scale)
  • question type indicators (template / model-generated)
  • safety flag
  • semantic annotations

Dataset Construction (Summary)

According to the methodology described in the paper:

1. Sourcing: Content was sourced from standard anatomy textbooks and clinical case reports.
2. Generation: Two pipelines generated QA pairs:
   • Rule-based templates
   • LLM-generated natural phrasing
3. Filtering: Safety filtering removed harmful, misleading, or clinically inappropriate questions.
4. Validation: Three licensed clinical annotators validated correctness and factual grounding.
   • Cohen’s Kappa agreement: 0.81 (indicating strong inter-annotator consistency).
5. Final Size: 1,077 validated QA pairs.

See Section 3 (Methodology) of the paper for full details.

Intended Use

This dataset is ideal for evaluating:

• Clinical question-answering models
• Factual alignment / hallucination resistance
• Safety-oriented model behaviors
• Reasoning on anatomical concepts
• Few-shot vs zero-shot performance gaps

The paper also benchmarks three open-source models:

• Mistral-7B
• BioMistral-7B-DARE
• AlpaCare-13B

Note: AlpaCare-13B achieved 91.7% accuracy and 0.92 harmlessness in the original study.

Safety Notice

Warning: This dataset is not intended to train clinical decision-making systems for real-world deployment without human oversight. It evaluates research models—not medical devices.

Citation

@inproceedings{azeez-etal-2025-truth,
    title = "Truth, Trust, and Trouble: Medical {AI} on the Edge",
    author = "Azeez, Mohammad Anas  and
      Ali, Rafiq  and
      Shabbir, Ebad  and
      Siddiqui, Zohaib Hasan  and
      Kashyap, Gautam Siddharth  and
      Gao, Jiechao  and
      Naseem, Usman",
    editor = "Potdar, Saloni  and
      Rojas-Barahona, Lina  and
      Montella, Sebastien",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = nov,
    year = "2025",
    address = "Suzhou (China)",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-industry.69/",
    doi = "10.18653/v1/2025.emnlp-industry.69",
    pages = "1017--1025",
    ISBN = "979-8-89176-333-3",
    abstract = "Large Language Models (LLMs) hold significant promise for transforming digital health by enabling automated medical question answering. However, ensuring these models meet critical industry standards for factual accuracy, usefulness, and safety remains a challenge, especially for open-source solutions. We present a rigorous benchmarking framework via a dataset of over 1,000 health questions. We assess model performance across honesty, helpfulness, and harmlessness. Our results highlight trade-offs between factual reliability and safety among evaluated models{---}Mistral-7B, BioMistral-7B-DARE, and AlpaCare-13B. AlpaCare-13B achieves the highest accuracy (91.7{\%}) and harmlessness (0.92), while domain-specific tuning in BioMistral-7B-DARE boosts safety (0.90) despite smaller scale. Few-shot prompting improves accuracy from 78{\%} to 85{\%}, and all models show reduced helpfulness on complex queries, highlighting challenges in clinical QA. Our code is available at: https://github.com/AnasAzeez/TTT"
}