license: cc-by-4.0
task_categories:

• text-classification
• text-generation
  language:
• en
  tags:
• safety
• red-teaming
• jailbreak
• llm-agent
• mcp
• benchmark
  pretty_name: FragBench
  version: "2.0.0"
  size_categories:
• 1K<n<10K
  citation: |
  @inproceedings{fragbench2027,
  title = {FragBench: A Benchmark for Detecting Fragmented Multi-Session Jailbreak Attacks on LLM Agents},
  author = {Anonymous},
  booktitle = {International Conference on Learning Representations (under review)},
  year = {2027}
  }
  configs:
• config_name: attack_meta
  data_files:
  • split: train
    path: "data/attack_meta.parquet"
    default: true
• config_name: campaign_fragments
  data_files:
  • split: train
    path: "data/campaign_fragments.parquet"

FragBench (Public Tier)

Anonymous submission to ICLR 2027.
Author identity will be revealed at camera-ready.

Dataset Summary

FragBench is a benchmark for evaluating cross-session, fragmented attacks on
LLM agents that use tools via the Model Context Protocol (MCP). Each campaign
is decomposed into many small fragments distributed across sessions; a
defender must reconstruct the compositional intent. The public tier in this
repository contains:

• Seed campaigns — the source malicious objectives, drawn from public
  cyber-incident reports
• Generated variants — stylised fragment sequences produced by the
  FragBench Attack pipeline
• Benign data — executed benign MCP tool-use traces used for negative
  examples

The more sensitive components of the FragBench Attack pipeline — reinforcement-learning
system prompts used by the attack rewriter, judge-rubric configurations and elicitation
scaffolds, and high-yield variant traces selected by the RL loop — are released separately
under request-access at the gated companion dataset:
anon-fragbench-neurips/fragbench-restricted

Supported Tasks

• Cross-session jailbreak detection: classify whether a session graph
  reconstructs a malicious campaign
• Multi-turn red-team evaluation: measure judge-pass rate of fragmented
  attacks on target LLMs
• Compositional safety: study aggregation features over typed
  session-event graphs

Languages

English.

Dataset Structure

The public tier is organised into the following top-level folders:

fragbench/
├── data/                     # Parquet rollups (used by the HF dataset viewer)
│   ├── attack_meta.parquet           # one row per attack pipeline run (41 rows)
│   └── campaign_fragments.parquet    # one row per fragment (~45k rows)
├── malicious/                # Malicious campaign data (positive class)
│   ├── attacks/                       # Per-run malicious attack outputs (JSON)
│   │                                  # naming: attack_<timestamp>_<runhash>_seed_N_<campaign>_N.json
│   │                                  # plus one *_meta.json per run with run-level config
│   ├── campaign-fragments/            # Per-campaign malicious fragment definitions
│   │                                  # one <campaign>_manual.json per campaign (24 files)
│   └── sessions/                      # Per-campaign malicious session traces, one per campaign
│                                      # (24 campaigns, ~600 sessions each, 15,055 total)
├── benign/                   # Executed benign control traces (negative class)
│   ├── attacks/                       # 514 chains + 514 graphs + 146 *_meta.json
│   ├── campaign-fragments/            # one benign_<campaign>_manual.json (24 files)
│   └── sessions/                      # one subfolder per campaign
│                                      # (24 campaigns, 5,440 sessions total)
└── combined/                 # Flat rollups for easy ingestion
    ├── malicious.json                # all malicious fragments, single file (~155 MB)
    └── benign.json                   # all benign sessions, single file (~155 MB)

Recommended entry points

• Quick exploration in the HF viewer → data/ (parquet)
• Single-file consumption for training a detector → combined/malicious.json + combined/benign.json
• Per-campaign analysis → malicious/campaign-fragments/<campaign>_manual.json
• Reproducing attack runs → malicious/attacks/ together with attack_meta.parquet
• Negative / benign baselines → combined/benign.json, or benign/ for the
  per-run, per-session and per-campaign files the rollup was built from

Fragment schema (key fields)

• campaign_id, variation_id, style
• fragments[]: ordered list of {prompt, expected_tool_calls, response}
• attack_graph: typed event graph with edges of type
  data_flow / temporal / shared_resource / shared_session / argument_similarity
• verdict: per-fragment LLM-judge label
  (ANSWERED / REFUSED / PARTIAL / UNCLEAR)

See the paper appendix for the full schema.

Dataset Creation

Source campaigns were transcribed from public cyber-incident reports. Each
seed was rewritten through a bootstrap → learned → creative loop using a
frontier LLM, with a single-token classifier-mode judge selecting variants
that increase pass rate while preserving the original objective.

Benign controls come from 24 benign campaign specifications authored by Claude
Opus 5, covering ordinary IT and office work. Seed definitions are expanded from
these specifications deterministically, with no model calls at expansion time,
then executed across six styles and nine random seeds in the same sandboxed MCP
harness as the malicious objectives.

Considerations for Using the Data

Intended uses

• Academic research on LLM-agent safety, multi-turn red-teaming, and
  cross-session attack detection
• Benchmarking detectors and guardrails on compositional jailbreaks

Out-of-scope / prohibited uses

• Direct deployment as offensive tooling against production systems
• Targeting individuals, organisations, or infrastructure without
  authorisation
• Use that violates the attribution requirements of CC-BY-4.0

Risks and mitigations

• Dual-use: prompts can elicit harmful outputs from underlying LLMs.
  The most sensitive RL system prompts are placed behind the gated tier.
• Bias: campaigns over-represent LLM-confirmed incidents disclosed by
  frontier-API vendors; coverage of non-frontier attacker ecosystems is
  limited.
• Sandbox-only: all execution traces in this dataset come from a
  sandboxed harness; no real third-party systems were targeted.

Licensing Information

This dataset is released under
CC-BY-4.0.
See LICENSE in this repository.

Citation

@inproceedings{fragbench2027,
  title  = {FragBench: A Benchmark for Detecting Fragmented Multi-Session
            Jailbreak Attacks on LLM Agents},
  author = {Anonymous},
  booktitle = {International Conference on Learning Representations (under review)},
  year   = {2027}
}