pretty_name: ART-SafeBench v2.0.0
license: cc-by-4.0
language:

• en
  multilinguality: monolingual
  size_categories:
• 10K<n<100K
  source_datasets:
• original
• HarmBench
• JailbreakBench
• Gandalf
• InjecAgent
• RAGuard
• TensorTrust
• ToolEmu
• FigStep
• AgentHarm (research only)
• Mindgard (research only)
• MM-SafetyBench (research only)
  tags:
• rag
• safety
• red-teaming
• adversarial
• multimodal
• agentic
• unified-benchmark
  task_categories:
• text-retrieval
• text-generation
• other
  dataset_creators:
• Inderjeet Singh (Fujitsu Research of Europe Limited, UK)
• Vikas Pahuja (Fujitsu Research of Europe Limited, UK)
• Aishvariya Priya Rathina Sabapathy (Fujitsu Research of Europe Limited, UK)
  configs:
  • config_name: B1_rag_text_poisoning
    data_files:
    • split: train
      path: data/attack_types/rag_poisoning_benchmark_combined_deduplicated.jsonl
  • config_name: B2_image_poisoning
    data_files:
    • split: train
      path: data/attack_types/image_poisoning_simulation_results_20250504_202954.jsonl
  • config_name: B3_direct_query
    data_files:
    • split: train
      path: data/attack_types/safety_benchmark_direct_query_combined_deduplicated.jsonl
  • config_name: B4_orchestrator
    data_files:
    • split: train
      path: data/attack_types/orchestrator_attacks_combined_deduplicated.jsonl
  • config_name: B1_augmented_raguard
    data_files:
    • split: train
      path: data/external_augmented/core/B1_augmented_raguard.jsonl
  • config_name: B2_augmented_figstep
    data_files:
    • split: train
      path: data/external_augmented/core/B2_augmented_figstep.jsonl
  • config_name: B3_augmented_harmbench
    data_files:
    • split: train
      path: data/external_augmented/core/B3_augmented_harmbench.jsonl
  • config_name: B3_augmented_jailbreakbench
    data_files:
    • split: train
      path: data/external_augmented/core/B3_augmented_jailbreakbench.jsonl
  • config_name: B3_augmented_gandalf
    data_files:
    • split: train
      path: data/external_augmented/core/B3_augmented_gandalf.jsonl
  • config_name: B4_augmented_injecagent
    data_files:
    • split: train
      path: data/external_augmented/core/B4_augmented_injecagent.jsonl
  • config_name: B4_augmented_toolemu
    data_files:
    • split: train
      path: data/external_augmented/core/B4_augmented_toolemu.jsonl

default_config_name: B1_rag_text_poisoning

Mirror note: This dataset is a mirror of Fujitsu/agentic-rag-redteam-bench, maintained by the same author, intherejeet, to preserve availability if organization access is interrupted. Access controls and usage restrictions are intended to match the source dataset.

ART-SafeBench v2.0.0

Overview

Version 2.0.0 preserves the original 36,192-record benchmark and ships +5,623 Core augmented records (redistributable), for 41,815 records in-package. It also provides runtime adapters for restricted-license datasets; when generated locally (and not redistributed), this yields 41,991+ total records (counts vary for some gated/multi-config sources). Additional adapters are included for further sources (see docs/master_data_inventory.md).

Dataset Statistics (Shipped vs. Reconstructable)

What's New in v2.0.0

• +5,623 Core Records Shipped across additional redistributable datasets (MIT/BSD/Apache)
• +Extended Records via Runtime for restricted-license datasets (do not redistribute)
• Unified Benchmark Design: Schema-aligned adapters for consistent external dataset integration
• Core + Extended Split: Clear separation between redistributable (Core) and research-only (Extended) datasets
• Production-Grade Adapters: Schema-compliant transformation scripts for each external source
• Unified B1-B4 Schema: All external data maps to the canonical attack surface taxonomy

Dataset Architecture

ART-SafeBench-v2.0.0/
├── data/
│   ├── attack_types/                    # Original v1.0.0 data (symlinked or copied)
│   └── external_augmented/
│       ├── core/                        # Redistributable augmented data
│       │   ├── B1_augmented_raguard.jsonl
│       │   ├── B2_augmented_figstep.jsonl
│       │   ├── B3_augmented_jailbreakbench.jsonl
│       │   ├── B3_augmented_gandalf.jsonl
│       │   ├── B3_augmented_harmbench.jsonl
│       │   ├── B4_augmented_injecagent.jsonl
│       │   ├── B4_augmented_toolemu.jsonl
│       │   └── figstep_images/           # FigStep typographic prompts (MIT)
│       └── extended/                    # Research-only (generated at runtime; not redistributed)
│           └── README.md
├── scripts/
│   ├── adapters/                        # Individual dataset adapters
│   │   ├── base_adapter.py
│   │   ├── b1_raguard_adapter.py
│   │   ├── b2_figstep_adapter.py
│   │   ├── b2_mmsafetybench_adapter.py
│   │   ├── b3_harmbench_adapter.py
│   │   ├── b3_jailbreakbench_adapter.py
│   │   ├── b3_gandalf_adapter.py
│   │   ├── b3_mindgard_adapter.py
│   │   ├── b3_tensortrust_adapter.py
│   │   ├── b4_injecagent_adapter.py
│   │   ├── b4_toolemu_adapter.py
│   │   └── b4_agentharm_adapter.py
│   ├── utils/
│   └── run_all_adapters.py              # Orchestration script
├── schema/                              # JSON schemas for all attack types
├── docs/                                # Extended documentation
└── reports/                             # Analysis and visualizations

External Dataset Inventory

Core Datasets (Safe to Redistribute - MIT/BSD/Apache)

Extended Datasets (Research Only - DO NOT REDISTRIBUTE)

See docs/master_data_inventory.md for a status-tracked inventory (implemented vs. planned sources).

Quick Start

Prerequisites

pip install -r scripts/requirements.txt

Running Core Adapters (Auto-Download)

cd scripts

# List available adapters
python run_all_adapters.py --list

# Run all Core adapters (auto-downloads + manual paths when provided)
python run_all_adapters.py --core-only

# Run specific adapter with record limit (for testing)
python run_all_adapters.py --adapter gandalf --max-records 1000

Running Extended Adapters (Research Only)

# WARNING: Extended datasets have usage restrictions
# Do NOT commit outputs to version control

python run_all_adapters.py --extended-only

Manual Download Datasets

Some datasets require manual download:

1. HarmBench: Clone repo, locate data/behavior_datasets/harmbench_behaviors_text_all.csv
2. InjecAgent: Clone repo, locate data/ directory
3. ToolEmu: Clone repo, locate assets/all_cases.json
4. FigStep: Clone repo, locate data/question/ and data/images/
5. TensorTrust: Obtain an exported raw_dump_attacks.jsonl (see upstream pipeline docs)

# Option A: Run via the orchestrator (recommended)
python run_all_adapters.py --core-only \
    --harmbench-csv-path /path/to/harmbench_behaviors_text_all.csv \
    --injecagent-data-dir /path/to/InjecAgent/data

# Additional manual adapters
python run_all_adapters.py --core-only \
    --toolemu-repo-dir /path/to/ToolEmu \
    --figstep-repo-dir /path/to/FigStep \
    --tensortrust-attacks-jsonl-path /path/to/raw_dump_attacks.jsonl

# Option B: Run individual adapters
python -m adapters.b3_harmbench_adapter --csv-path /path/to/harmbench_behaviors_text_all.csv
python -m adapters.b4_injecagent_adapter --data-dir /path/to/InjecAgent/data
python -m adapters.b4_toolemu_adapter --repo-dir /path/to/ToolEmu
python -m adapters.b2_figstep_adapter --repo-dir /path/to/FigStep
python -m adapters.b3_tensortrust_adapter --attacks-jsonl-path /path/to/raw_dump_attacks.jsonl

One-Command Reconstruction (Recommended for Users)

For a fully reproducible setup (including cloning public Core sources), run:

cd scripts
python reconstruct_external_data.py --core --clone-core-sources --figstep-copy-images

# Research-only (do not redistribute):
python reconstruct_external_data.py --extended

Attack Surface Mapping

B1: RAG Text Poisoning (Knowledge Store)

• RAGuard: Misleading/irrelevant retrieval scenarios
• Chain: user_query → retrieval returns poisoned text → Generator consumes → violation

B2: Image Poisoning (OCR-based)

• FigStep (SafeBench): Typographic image prompts + benign completion prompt (adapter)
• MM-SafetyBench (Extended): Multimodal safety prompts with images (adapter; research only)
• Chain: user_query → retrieval includes poisoned image → OCR → Generator

B3: Direct Query Attacks (User Input)

• HarmBench: Curated harmful behaviors
• JailbreakBench: Jailbreak prompts with category labels
• Gandalf: Human-generated prompt injections
• Mindgard (Extended): Evaded guardrail samples
• Chain: adversarial user_query → direct LLM attack

B4: Orchestrator Attacks (Agent/Tool Layer)

• InjecAgent: Indirect prompt injection in tool contexts
• AgentHarm (Extended): Malicious agent behaviors
• Chain: benign_query + injection → tool flip/argument manipulation

Schema Compliance

All adapters produce JSONL records compliant with the v1.0.0 schemas:

• schema/rag_poisoning.schema.json (B1)
• schema/image_poisoning.schema.json (B2)
• schema/safety_direct_query.schema.json (B3)
• schema/orchestrator_attacks.schema.json (B4)

Additional fields for provenance tracking:

• source_dataset: Origin dataset name
• source_license: License of source dataset
• source_id: Original record identifier (if available)
• research_only: Boolean flag for restricted datasets

Loading Augmented Data

from datasets import load_dataset
from pathlib import Path

# Load Core augmented data
core_dir = Path("data/external_augmented/core")

for jsonl_file in core_dir.glob("*.jsonl"):
    ds = load_dataset("json", data_files=str(jsonl_file), split="train")
    print(f"{jsonl_file.name}: {len(ds)} records")
    print(f"  Fields: {list(ds.features.keys())[:5]}...")

Combined Evaluation Pipeline

import json
from pathlib import Path
from collections import defaultdict

def load_all_benchmarks(base_dir: Path, include_extended: bool = False):
    """Load all benchmark data, optionally including Extended datasets."""
    data_by_surface = defaultdict(list)

    # Load original v1.0.0 data
    original_dir = base_dir / "data" / "attack_types"
    for f in original_dir.glob("*.jsonl"):
        with open(f) as fp:
            for line in fp:
                record = json.loads(line)
                # Infer surface from filename
                if "rag_poisoning" in f.name:
                    data_by_surface["B1"].append(record)
                elif "image_poisoning" in f.name:
                    data_by_surface["B2"].append(record)
                elif "direct_query" in f.name:
                    data_by_surface["B3"].append(record)
                elif "orchestrator" in f.name:
                    data_by_surface["B4"].append(record)

    # Load Core augmented data
    core_dir = base_dir / "data" / "external_augmented" / "core"
    for f in core_dir.glob("*.jsonl"):
        surface = f.name.split("_")[0]  # B1, B2, B3, or B4
        with open(f) as fp:
            for line in fp:
                data_by_surface[surface].append(json.loads(line))

    # Optionally load Extended data
    if include_extended:
        ext_dir = base_dir / "data" / "external_augmented" / "extended"
        for f in ext_dir.glob("*.jsonl"):
            surface = f.name.split("_")[0]
            with open(f) as fp:
                for line in fp:
                    data_by_surface[surface].append(json.loads(line))

    return data_by_surface

# Usage
benchmark_data = load_all_benchmarks(Path("."), include_extended=False)
for surface, records in benchmark_data.items():
    print(f"{surface}: {len(records)} total records")

Licensing Summary

IMPORTANT: Extended dataset outputs are generated at runtime and must NOT be committed to version control or redistributed.

Citation

@misc{art_safebench_v2_0_0_2025,
  title        = {ART-SafeBench},
  author       = {Singh, Inderjeet and Pahuja, Vikas and Rathina Sabapathy, Aishvariya Priya},
  year         = {2025},
  version      = {2.0.0},
  howpublished = {Dataset},
  license      = {CC-BY-4.0},
  note         = {Augmented benchmark with unified external dataset adapters}
}

Authors

• Inderjeet Singh (Fujitsu Research of Europe Limited, UK)
• Vikas Pahuja (Fujitsu Research of Europe Limited, UK)
• Aishvariya Priya Rathina Sabapathy (Fujitsu Research of Europe Limited, UK)

Contact

• General: [email protected]
• Responsible-use/safety: [email protected]

Acknowledgments

We gratefully acknowledge the creators of the external datasets integrated in this benchmark:

• Center for AI Safety (HarmBench)
• JailbreakBench Team
• Lakera (Gandalf)
• UIUC Kang Lab (InjecAgent)
• UCSC IRKM (RAGuard)
• HumanCompatibleAI (TensorTrust / PromptGame)
• ToolEmu authors (Ruan et al.)
• ThuCCSLab (FigStep)
• UK AI Safety Institute / Gray Swan AI (AgentHarm)
• Mindgard