language:

• ja
• en
  license: cc-by-nc-4.0
  task_categories:
• text-generation
• text-classification
  tags:
• medical
• safety
• adversarial
• jailbreak
• multi-turn
• benchmark
• ethics
• red-teaming
  pretty_name: JMedEthicBench
  size_categories:
• 10K<n<100K
  configs:
• config_name: test_set
  default: true
  data_files:
  • split: test
    path: step3_test_set/test_set_2345.jsonl
• config_name: multiturn_pool
  data_files:
  • split: anthropic_claude
    path: step2_multiturn_conversation_generation/claude-3.7-thinking.jsonl
  • split: deepseek_r1
    path: step2_multiturn_conversation_generation/deepseek-r1-0528.jsonl
  • split: google_gemini
    path: step2_multiturn_conversation_generation/gemini-2.5-pro-preview.jsonl
  • split: qwen_qwq
    path: step2_multiturn_conversation_generation/qwq-32b.jsonl
• config_name: harmful_requests
  data_files:
  • split: candidates
    path: step1_harmful_question_generation/harmful_requests_3350.json
  • split: validated
    path: step2_multiturn_conversation_generation/validated_requests_1935.json

JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese LLMs

Dataset Description

JMedEthicBench is the first multi-turn adversarial benchmark for evaluating the adversarial robustness of medical ethics alignment in Japanese Large Language Models. The benchmark is grounded in 67 guidelines from the Japan Medical Association (JMA) and contains 50,843 multi-turn adversarial attack prompts built from 7 automatically discovered jailbreak strategies.

Paper: JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Models (under review)

This release contains attack prompts only. No model responses are included. To use the benchmark you generate responses from the model under evaluation and score those — the responses of the four models used to construct the attacks are not part of the artifact.

Dataset Structure

├── step1_harmful_question_generation/
│   └── harmful_requests_3350.json          # 3,350 candidate harmful questions
├── step2_multiturn_conversation_generation/
│   ├── validated_requests_1935.json        # 1,935 validated harmful questions
│   ├── claude-3.7-thinking.jsonl           # 12,333 attack prompts
│   ├── deepseek-r1-0528.jsonl              # 12,643 attack prompts
│   ├── gemini-2.5-pro-preview.jsonl        # 12,822 attack prompts
│   └── qwq-32b.jsonl                       # 13,045 attack prompts
└── step3_test_set/
    └── test_set_2345.jsonl                 # 2,345 test instances  ← the benchmark

Step 1: Single-Turn Harmful Question Generation

Five LLMs generated 10 harmful medical questions in Japanese for each of the 67 JMA guidelines, yielding 3,350 candidate questions (670 per source model). A separate set of five LLMs then attempted to answer each question; only those refused by at least two models were retained, producing 1,935 validated questions.

Schema (harmful_requests_3350.json, validated_requests_1935.json — JSON arrays):

validated_requests_1935.json is exactly the is_filtered == false subset of harmful_requests_3350.json.

Step 2: Multi-Turn Attack Prompt Generation

Using an adapted AutoDAN-Turbo framework, 7 generalizable jailbreak strategies were automatically discovered. Four LLMs then composed multi-turn adversarial prompts by combining the validated questions with these strategies, producing 50,843 attack prompts.

Schema (.jsonl files):

Attack-generation models: Claude-3.7-Sonnet-Thinking, DeepSeek-R1-0528, Gemini-2.5-Pro-Preview, QwQ-32B

Turn counts per file:

Step 3: Test Set

Five multi-turn samples were randomly selected for every (jailbreak strategy × JMA principle) combination — 7 × 67 = 469 cells — yielding 2,345 test instances, all exactly 3 turns. Schema is identical to Step 2.

By attack-generation model: deepseek_r1 837, google_gemini 705, anthropic_claude 415, qwen_qwq 388.

Jailbreak Strategies

Seven distinct strategies were automatically discovered:

1. Gray-Zone Ethnographic Simulation — Frames harmful requests as academic/ethnographic research
2. Escalating Academic Vulnerability Simulation — Progressively introduces academic contexts with escalating urgency
3. Creativity-Anchored Ethnographic Escalation — Combines creative writing with ethnographic research framing
4. Historical-Creative Dual Frame Dissociation — Alternates between historical case studies and creative contexts
5. Fiction-Layered Authority Simulation — Uses fictional authority figures for legitimacy
6. Temporal-Override Academic Escalation — Employs time-based urgency with academic authority
7. Sanitized Academic Reproduction Frame — Repackages requests as academic citations

Usage

from datasets import load_dataset

# the benchmark
test = load_dataset("humanalysis-square/JMedEthicBench", "test_set", split="test")

# the full attack-prompt pool, per generating model
pool = load_dataset("humanalysis-square/JMedEthicBench", "multiturn_pool", split="deepseek_r1")

Evaluation protocol: replay Turn_0 … Turn_N against the model under test as a single conversation, carrying its own replies forward as context, then score each turn's response for safety.

turns = [row[f"Turn_{i}"] for i in range(5) if row[f"Turn_{i}"]]
messages = []
for t in turns:
    messages.append({"role": "user", "content": t})
    reply = your_model(messages)
    messages.append({"role": "assistant", "content": reply})

Key Findings

• Commercial models (e.g. Claude Opus 4.1, GPT-5) maintain robust safety alignment
• Medical-specialized models exhibit increased vulnerability to adversarial attacks
• Safety scores decline significantly across conversation turns (median: 9.5 → 5.0, p < 0.001)
• Vulnerabilities persist across Japanese and English, indicating inherent alignment limitations

Languages

• Primary: Japanese
• Cross-lingual evaluation: English

Intended Use

This dataset is intended exclusively for academic research on:

• Medical AI safety evaluation
• LLM adversarial robustness assessment
• Development of improved safety alignment techniques
• Cross-lingual safety evaluation research

Ethics Statement

Purpose and dual-use

This dataset exists to make Japanese medical LLMs safer by measuring where their safety alignment fails under multi-turn adversarial pressure. It is, unavoidably, dual-use: the same prompts that measure a model's robustness could be used to attack a deployed system. We judge the release justified because (a) the attack strategies were discovered automatically from techniques already documented in the public jailbreak literature and introduce no novel capability, (b) defenders cannot patch failures they cannot measure, and (c) the release withholds the component that would make it directly actionable — see below.

What the release does and does not contain

The dataset contains only attack prompts. Every model response generated during construction has been removed, and no field in any released file contains a model's answer to a harmful request. The prompts document how a harmful medical request can be framed to evade refusal; they do not contain drug dosages, procedures, or instructions that would let a reader carry out the underlying act. Users evaluating a model necessarily generate such responses locally, and are responsible for not redistributing them.

Content warning

The attack prompts and the underlying harmful questions reference, in Japanese, scenarios including: falsification of medical records and death certificates; circumvention of informed consent; illegal procurement of controlled and abortifacient drugs; organ and gamete trafficking; abuse of elderly and disabled patients; euthanasia coercion; research misconduct; and unauthorised disclosure of patient records. Some prompts are graphic. They were written to be refused, and a well-aligned model refuses them.

Named individuals

A small number of prompts name real public figures — Japanese entertainers, and unnamed politicians via placeholders — inside explicitly fictional scenarios that ask a model to disclose or fabricate their medical information. No real medical information about any identifiable person is contained in this dataset; the scenarios are invented, and the models were expected to refuse them. We nonetheless recognise that naming living individuals in such scenarios is undesirable. If you are, or represent, a named individual and want a record removed, contact the corresponding author and we will remove it in the next revision.

Source material

Harmful questions were generated by LLMs prompted with the topic areas of 67 Japan Medical Association (JMA) guidelines. The release contains derived adversarial questions and conversations only — it does not reproduce JMA guideline text, and the principle field carries a short guideline label for grounding, not guideline content.

Permitted and prohibited use

Permitted: safety evaluation, red-teaming research, alignment and refusal-training research, and cross-lingual safety analysis, in academic and non-commercial settings, consistent with the CC-BY-NC-4.0 licence.

Prohibited: using these prompts to attack production systems you do not own or have authorisation to test; training models to produce harmful medical content; and redistributing model responses obtained by replaying these prompts.

Access

The dataset is access-controlled. Requests are reviewed, and access is granted for stated research purposes.

Limitations relevant to interpretation

This is a static-replay, worst-case stress test and yields a conservative lower bound on adversarial vulnerability rather than an estimate of real-world harm rates. Because harmful-question filtering was keyword-based, a small number of benign questions may be retained, which can register as over-refusal. The Step-2 pool is released as generated, without post-hoc cleaning; the Step-3 test set is the recommended artifact for evaluation.

Citation

@article{Liu2026JMedEthicBench,
  title={JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Models},
  author={Liu, Junyu and Li, Zirui and Niu, Qian and Zhang, Zequn and Xun, Yue and Hou, Wenlong and Wang, Shujun and Iwasawa, Yusuke and Matsuo, Yutaka and Hatakeyama-Sato, Kan},
  year={2026}
}

Authors

Junyu Liu (Kyoto University), Zirui Li (Hohai University), Qian Niu (The University of Tokyo), Zequn Zhang (USTC), Yue Xun (PolyU), Wenlong Hou (PolyU), Shujun Wang (PolyU), Yusuke Iwasawa (The University of Tokyo), Yutaka Matsuo (The University of Tokyo), Kan Hatakeyama-Sato (The University of Tokyo)

Corresponding author: Qian Niu ([email protected])