configs:
- config_name: refusal_reference_set
data_files:- split: train
path: refusal_reference_set.csv
- split: train
- config_name: experiment_results
data_files:- split: train
path: experiment_results.csv
- split: train
license: cc-by-4.0
tags:
- cybersecurity
- multi-turn
- adversarial-ai
- tense-reformulation
- jailbreaking-evaluation
- semantic-analysis
RefusalGuard-M Dataset
This repository contains the datasets associated with the RefusalGuard-M framework for multi-turn LLM jailbreak evaluation via semantic refusal
manifold modelling.
Associated Paper: RefusalGuard-M: A Scalable Human–Machine Framework for Multi-Turn LLM Jailbreak Evaluation via Semantic Refusal Manifold Modeling
Dataset Structure
| File | Description |
|---|---|
refusal_reference_set.csv |
Human-annotated refusal reference samples used to construct the refusal manifold |
experiment_results.csv |
Experimental results obtained from evaluating RefusalGuard-M |
Dataset Statistics
| Dataset | Size | Description |
|---|---|---|
| Refusal Reference Set | 1.01k samples | Human-annotated refusal samples |
| Experiment Results | 7.11k samples | Model evaluation results |
Column Descriptions For Refusal Reference Set
| Column Name | Description |
|---|---|
refusal |
Human-annotated refusal response used as a reference example for modelling the semantic refusal manifold. |
tense |
Grammatical tense condition associated with the corresponding conversation (present or past). |
subtopic |
Specific cybersecurity subtopic associated with the conversation, such as Hacking, Malware, or Phishing. |
Column Descriptions For Experiments Results
| Column Name | Description |
|---|---|
base_id |
Original prompt index (0–126). |
conversation_id |
Unique identifier for each conversation, formatted as base_id_turn_tense. |
turn_depth |
Number of turns in the conversation (1–3). |
tense |
Grammatical tense condition used in the conversation (present or past). |
semantic_category |
Broad cybersecurity category associated with the query, based on the AdvBench/HarmBench taxonomy. |
subtopic |
Specific cybersecurity topic covered by the query, such as Hacking, Malware, or Phishing. |
query_type |
Type of request: Instruction for procedural/how-to requests or Script for requests involving code/scripts. |
source |
Original data source, such as AdvBench, HarmBench, or Original. |
attack_name |
Name of the jailbreak attack or attack strategy used to generate the conversation (Mirage, Crescendo, or Tempest). |
turn_1 |
First user turn in the multi-turn conversation. |
output_turn_1 |
Target model's response to the first turn. |
turn_2 |
Second user turn in the conversation; empty when turn_depth < 2. |
output_turn_2 |
Target model's response to the second turn; empty when turn_depth < 2. |
turn_3 |
Third user turn in the conversation; empty when turn_depth < 3. |
output_turn_3 |
Target model's response to the third turn; empty when turn_depth < 3. |
human_i_judge |
Label assigned by the i-th human annotator. |
human_ensemble_judge |
Aggregated label derived from the human-annotator ensemble. |
semantic_score_<judge> |
Semantic score assigned by the specified automated judge, representing the degree of semantic alignment between the target-model response and the evaluation criterion. |
prob_jailbreak_<judge> |
Jailbreak probability score assigned by the specified automated judge to its classification. |
prediction_<judge> |
Label assigned by the specified automated judge to the target-model response. |
confidence_<judge> |
Confidence score assigned by the specified automated judge to its classification. |
Label Definition
0= Refusal1= Jailbreak
Judge Fields
Judge-specific fields follow the naming convention:
semantic_score_<judge>
prob_jailbreak_<judge>
prediction_<judge>
confidence_<judge>
Citation
If you find this dataset useful in your research, please consider citing:
@article{tchuindjang2026refusalguard-m,
author = {Tchuindjang, Michael and Duran, Nathan. and Legg, Phil. and Medjek, Faiza.},
title = {RefusalGuard-M: A scalable human--machine framework for multi-turn LLM jailbreak evaluation via semantic refusal manifold modeling},
journal = {Cybersecurity},
volume = {9},
pages = {206},
year = {2026},
doi = {10.1186/s42400-026-00633-z}
}