license: cc-by-4.0
task_categories:

• text-classification
• text-generation
  language:
• en
  tags:
• safety
• red-teaming
• jailbreak
• llm-agent
• mcp
• benchmark
  pretty_name: FragBench
  version: "1.0.0"
  size_categories:
• 1K<n<10K
  citation: |
  @inproceedings{fragbench2026,
  title = {FragBench: A Benchmark for Detecting Fragmented Multi-Session Jailbreak Attacks on LLM Agents},
  author = {Anonymous},
  booktitle = {NeurIPS Datasets and Benchmarks Track (under review)},
  year = {2026}
  }
  configs:
• config_name: attack_meta
  data_files:
  • split: train
    path: "data/attack_meta.parquet"
    default: true
• config_name: campaign_fragments
  data_files:
  • split: train
    path: "data/campaign_fragments.parquet"

FragBench (Public Tier)

Anonymous submission for NeurIPS 2026 Datasets and Benchmarks Track.
Author identity will be revealed at camera-ready.

Dataset Summary

FragBench is a benchmark for evaluating cross-session, fragmented attacks on
LLM agents that use tools via the Model Context Protocol (MCP). Each campaign
is decomposed into many small fragments distributed across sessions; a
defender must reconstruct the compositional intent. The public tier in this
repository contains:

• Seed campaigns — the source malicious objectives, drawn from public
  cyber-incident reports
• Generated variants — stylised fragment sequences produced by the
  FragBench Attack pipeline
• Benign data — synthetic benign tool-use traces used for negative
  examples

The more sensitive components of the FragBench Attack pipeline — reinforcement-learning
system prompts used by the attack rewriter, judge-rubric configurations and elicitation
scaffolds, and high-yield variant traces selected by the RL loop — are released separately
under request-access at the gated companion dataset:
anon-fragbench-neurips/fragbench-restricted

Supported Tasks

• Cross-session jailbreak detection: classify whether a session graph
  reconstructs a malicious campaign
• Multi-turn red-team evaluation: measure judge-pass rate of fragmented
  attacks on target LLMs
• Compositional safety: study aggregation features over typed
  session-event graphs

Languages

English.

Dataset Structure

The public tier is organised into the following top-level folders:

fragbench/
├── data/                     # Parquet rollups (used by the HF dataset viewer)
│   ├── attack_meta.parquet           # one row per attack pipeline run (41 rows)
│   └── campaign_fragments.parquet    # one row per fragment (~45k rows)
├── attacks/                  # Per-run attack outputs (JSON)
│                             # naming: attack_<timestamp>_<runhash>_seed_N_<campaign>_N.json
│                             # plus one *_meta.json per run with run-level config
├── campaign-fragments/       # Per-campaign manual fragment definitions
│                             # one <campaign>_manual.json per campaign (24 files)
├── combined/                 # Flat rollups for easy ingestion
│   ├── malicious.json                # all malicious fragments, single file (~155 MB)
│   └── benign.json                   # all benign sessions, single file (~50 MB)
├── benign/                   # Synthetic benign session chains (jsonl)
│                             # naming: session_chain_<chain_id>_s<step>.jsonl
└── sessions/                 # Per-campaign session traces, one subfolder per campaign
                              # (24 campaigns, ~40 sessions each)

Recommended entry points

• Quick exploration in the HF viewer → data/ (parquet)
• Single-file consumption for training a detector → combined/malicious.json + combined/benign.json
• Per-campaign analysis → campaign-fragments/<campaign>_manual.json
• Reproducing attack runs → attacks/ together with attack_meta.parquet
• Negative / benign baselines → benign/ (jsonl) or combined/benign.json

Fragment schema (key fields)

• campaign_id, variation_id, style
• fragments[]: ordered list of {prompt, expected_tool_calls, response}
• attack_graph: typed event graph with edges of type
  data_flow / temporal / shared_resource / shared_session / argument_similarity
• verdict: per-fragment LLM-judge label
  (ANSWERED / REFUSED / PARTIAL / UNCLEAR)

See the paper appendix for the full schema.

Dataset Creation

Source campaigns were transcribed from public cyber-incident reports. Each
seed was rewritten through a bootstrap → learned → creative loop using a
frontier LLM, with a single-token classifier-mode judge selecting variants
that increase pass rate while preserving the original objective.

Considerations for Using the Data

Intended uses

• Academic research on LLM-agent safety, multi-turn red-teaming, and
  cross-session attack detection
• Benchmarking detectors and guardrails on compositional jailbreaks

Out-of-scope / prohibited uses

• Direct deployment as offensive tooling against production systems
• Targeting individuals, organisations, or infrastructure without
  authorisation
• Use that violates the attribution requirements of CC-BY-4.0

Risks and mitigations

• Dual-use: prompts can elicit harmful outputs from underlying LLMs.
  The most sensitive RL system prompts are placed behind the gated tier.
• Bias: campaigns over-represent LLM-confirmed incidents disclosed by
  frontier-API vendors; coverage of non-frontier attacker ecosystems is
  limited.
• Sandbox-only: all execution traces in this dataset come from a
  sandboxed harness; no real third-party systems were targeted.

Licensing Information

This dataset is released under
CC-BY-4.0.
See LICENSE in this repository.

Citation

@inproceedings{fragbench2026,
  title  = {FragBench: A Benchmark for Detecting Fragmented Multi-Session
            Jailbreak Attacks on LLM Agents},
  author = {Anonymous},
  booktitle = {NeurIPS Datasets and Benchmarks Track (under review)},
  year   = {2026}
}