license: cc-by-nc-4.0
task_categories:

• text-generation
  language:
• en
  tags:
• safety
• dpo
• preference-optimization
• mcp
• model-context-protocol
• agentic
• red-teaming
• jailbreak
• refusal
  pretty_name: MCP Falsely-Benign Attack (FBA) & Truly-Benign (TB) Preference Dataset
  size_categories:
• 1K<n<10K
  gated: true
  extra_gated_heading: Acknowledge terms to access this dataset
  extra_gated_prompt: This dataset contains CVE-derived MCP attack examples for defensive
  safety research only (training/evaluating refusal behavior in tool-using LLM agents).
  By requesting access, you agree not to use it to train, fine-tune, or otherwise
  improve a model for carrying out, facilitating, or evading detection of malicious
  attacks against real systems.
  extra_gated_fields:
  Company/Affiliation: text
  Intended use:
  type: select
  options:
  • Academic research
  • Industry safety research
  • Model evaluation / benchmarking
  • label: Other
    value: other
    I agree to use this dataset only for defensive safety research, not to train a model for malicious purposes: checkbox
    extra_gated_button_content: Agree and request access
    configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*
• config_name: test_attack
  data_files:
  • split: test
    path: test_attack/test-*
• config_name: test_benign
  data_files:
  • split: test
    path: test_benign/test-*

MCP Falsely-Benign Attack (FBA) & Truly-Benign (TB) Preference Dataset

TL;DR

Most LLM safety training targets prompts that look malicious. This dataset targets prompts
that don't. It contains preference pairs for training refusal guardrails against
falsely benign attacks (FBAs) — Model Context Protocol (MCP) tool-use exploits derived
from real CVEs, phrased as ordinary, harmless-sounding requests with no refusal-triggering
language, paired with truly-benign (TB) MCP tool-use requests for contrast.

No safety-tuned base model in the source study exceeded a 35% refusal rate on these attacks
out of the box, and even after DPO/SafeDPO alignment, no model crossed 48%. Training and
evaluation code for four methods — DPO, SafeDPO, OPAD, and the training-free RAG-Pref — is
available in the companion repo.

Splits

test_attack columns

Each row traces one CVE through the full generation pipeline, from raw vulnerability
to the natural-language prompt used to test the model:

cveId + description  ──▶  commands  ──▶  mcp_commands  ──▶  prompt
 (the CVE)              (Linux steps)   (MCP tool calls)   (attack prompt)

test_benign rows contain only a prompt column, since they were authored directly
as legitimate requests rather than derived from a CVE pipeline.

How the attacks were built

Attacks were sourced from the CVE catalog and filtered to exploits achievable through
10 Linux-style filesystem MCP tools (read_file, write_file, edit_file,
move_file, etc.). Each feasible CVE was mapped to a sequence of MCP tool calls, then
rewritten into a "friendly" natural-language request that avoids phrasing which would commonly trigger
safety guardrails — hence falsely benign. Truly-benign samples were generated
by prompting Claude to produce legitimate MCP tool-use requests across a range of
personas (sysadmin, researcher, student, etc.), hand-verified, with dispreferred
responses created by substituting in the wrong (but still benign-sounding) tool call.

Why this matters

Standard safety alignment (DPO, SafeDPO) only improved refusal rates on these attacks by
an average factor of 1.4 and 1.6 respectively, and no evaluated model — from 1B to 14B
parameters — reached even a 50% refusal rate. RAG-Pref, a training-free, retrieval-based
alignment approach introduced alongside this data, achieved roughly a 3-fold improvement
in refusal rates over baseline models, and combined with DPO/SafeDPO reached an average
3.7-fold improvement — all without the GPU cost of fine-tuning. Full training and
evaluation code for all four methods is in the companion repo linked above.

If you're training or evaluating refusal guardrails for tool-using / agentic LLMs, this dataset
targets a gap that standard jailbreak/harmful-prompt datasets don't cover: attacks with
no lexical red flags at all.

Usage

This dataset has three configs (not just splits) since the training data and the
eval sets have different schemas:

from datasets import load_dataset

train_ds = load_dataset("johnhalloran/mcp-fbas", "default", split="train")   # chosen/rejected pairs
attack_ds = load_dataset("johnhalloran/mcp-fbas", "test_attack", split="test")  # FBA eval prompts
benign_ds = load_dataset("johnhalloran/mcp-fbas", "test_benign", split="test")  # TB eval prompts

Training example (adapted from the training code used for this dataset — 4-bit QLoRA + DPO via TRL):

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

from trl import DPOConfig, DPOTrainer, ModelConfig, get_peft_config, setup_chat_format

model_name_or_path = "meta-llama/Llama-3.2-1B-Instruct"  # or your model of choice

# 4-bit quantized base model
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_name_or_path,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    quantization_config=bnb_config,
)
ref_model = None  # not needed when training with a PEFT adapter

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
if tokenizer.chat_template is None:
    model, tokenizer = setup_chat_format(model, tokenizer)
if tokenizer.model_max_length > 100_000:
    tokenizer.model_max_length = 1024

# Load this dataset's train split (FBA + TB preference pairs)
dataset = load_dataset("johnhalloran/mcp-fbas", "default", split="train")

training_args = DPOConfig(
    output_dir="dpo-fba-tb-model",
    max_length=1024,
    max_prompt_length=512,
    save_strategy="no",
)

model_args = ModelConfig(model_name_or_path=model_name_or_path)
peft_config = get_peft_config(model_args)
peft_config.target_modules = [
    "q_proj", "k_proj", "v_proj", "o_proj", "up_proj", "down_proj", "gate_proj",
]

trainer = DPOTrainer(
    model,
    ref_model,
    args=training_args,
    train_dataset=dataset,
    processing_class=tokenizer,
    peft_config=peft_config,
)
trainer.train()
trainer.save_model(training_args.output_dir)
tokenizer.save_pretrained(training_args.output_dir)

See the full training script (with CLI args, HfArgumentParser, and Hub push logic),
plus SafeDPO, OPAD, and RAG-Pref implementations, in the companion code repo:
https://github.com/johnhalloran321/mcp_safety_training

Limitations & responsible use

• FBA prompts encode real (if reworded) attack patterns; this data is intended for
  defensive alignment research — training and evaluating refusal behavior — not for
  crafting new attacks.
• Refusal quality was judged using an automated two-stage classifier pipeline in the
  source study, not exhaustive human review of every generation; treat reported refusal
  rates as approximate.
• MCP tool coverage is limited to filesystem-style tools; results may not generalize to
  other tool categories (network, shell exec, browser, etc.) without further data.

Citation

If you use this dataset, please cite the original research it's derived from:

@article{halloran2025mcpsafety,
  title   = {MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignment},
  author  = {Halloran, John},
  journal = {arXiv preprint arXiv:2505.23634},
  year    = {2025}
}

@article{halloran2026ragpref,
  title   = {Leveraging RAG for Training-Free Alignment of LLMs},
  author  = {Halloran, John},
  journal = {arXiv preprint arXiv:2605.11217},
  year    = {2026}
}

License

CC BY-NC 4.0 — free to use, share, and adapt for research and non-commercial purposes,
with attribution. Commercial use is not permitted under this license; contact the
authors if you need different terms.