license: mit
task_categories:

• text-classification
  language:
• en
  tags:
• prompt-injection
• llm-security
• red-teaming
• benchmark
• ai-safety
• delimiter-defense
  size_categories:
• 1K<n<10K
  pretty_name: DataBoundary - Prompt Injection Defense Benchmark

DataBoundary: Prompt Injection Defense Benchmark

A red/blue team benchmark dataset evaluating delimiter-based defenses against prompt injection attacks across 13 LLMs, including both cloud APIs and locally-hosted models.

Overview

Models

Defense Rate = PASS / (PASS + FAIL), excluding errors.

Dataset Structure

Each record in full_results.jsonl contains:

Usage

from datasets import load_dataset

ds = load_dataset("Alan-StratCraftsAI/databoundary")

# Filter to successful results only
df = ds["train"].to_pandas()
clean = df[df["error"].isna()]

# Defense rate by model
rates = clean.groupby("model")["judgment"].apply(
    lambda x: (x == "PASS").sum() / len(x)
).sort_values(ascending=False)
print(rates)

Methodology

This benchmark uses a red/blue team approach:

• Red team: Crafts prompt injection attacks using multiple templates and payloads designed to override system instructions
• Blue team: Defends using delimiter-based boundary markers that separate trusted instructions from untrusted user content
• Judgment: Automated evaluation determines whether each model followed the injected instructions (FAIL) or maintained its original task (PASS)

For full methodology details, see the project repository.

Error Records

Some records contain API errors (timeouts, rate limits, content filtering). These are preserved with the error field populated. To analyze only successful completions:

df = df[df["error"].isna()]

Key Findings

• Claude models show the strongest delimiter defense (100% defense rate)
• GPT-5.4 Mini achieves 95.7% defense rate, significantly outperforming GPT-4o (86.6%)
• Local models (Gemma 4, Qwen 2.5 7B) show moderate defense rates (60-69%), lower than cloud counterparts
• Delimiter-based defenses are effective but not uniformly so — model architecture and training significantly impact robustness

Citation

@dataset{databoundary2026,
  title={DataBoundary: Prompt Injection Defense Benchmark},
  author={StratCraft AI},
  year={2026},
  url={https://huggingface.co/datasets/Alan-StratCraftsAI/databoundary},
  note={Red/blue team benchmark for delimiter-based prompt injection defenses}
}

Links

• Source Code: github.com/Alan-StratCraftsAI/DataBoundary
• Organization: StratCraft AI — a research lab exploring AI safety and applied ML

License

MIT