license: apache-2.0
tags:

• red-teaming
• llm-security
• owasp
• benchmark
• evaluation
• agent-evaluation
  task_categories:
• text-classification
  size_categories:
• 1K<n<10K
  language:
• en

ORQ Red-Teaming Framework Benchmark

Overview

This dataset contains the full results of a comparative red-teaming benchmark evaluating three
open-source red-teaming frameworks — EvaluatorQ, DeepTeam, and PromptFoo — against
three victim LLMs across three target configurations and five OWASP LLM Top 10 (2025) vulnerability
categories.

Each row is one attack attempt: the attack prompt sent to the victim model, the model's response,
and the verdict from a 3-model majority-vote panel of LLM judges.

Coverage

Total rows: 5,660 across 3 models, 3 frameworks, 3 targets, 5 OWASP categories.

Panel-vulnerable rate (all rows): 19.5%

By framework

By model

By target configuration

By OWASP category

Schema

Identity columns

Content columns

Panel judge columns

Run metadata columns

Panel judges

Verdicts are produced by a 3-model majority vote:

panel_vulnerable = True when ≥ 2 of 3 judges vote vulnerable. panel_votes stores the
per-judge boolean as a JSON-encoded dict ({"glm-5-maas": false, "claude-sonnet-4-6": false, "minimax-m2.5": true}).
Judge reasoning/explanation is not stored in this release (see Limitations).

OWASP LLM categories

Benchmarked (all 3 frameworks)

Defined but out of scope for this benchmark

Usage

from datasets import load_dataset

ds = load_dataset("orq/redteam-framework-benchmark", data_files="hf_dataset.jsonl", split="train")
print(ds)
# Dataset({features: [...], num_rows: 5,660})

# Filter to panel-vulnerable attacks on the secure agent
secure_vulns = ds.filter(lambda r: r["target"] == "secure-agent" and r["panel_vulnerable"])

Or with pandas:

import pandas as pd, json

df = pd.read_csv("hf://datasets/orq/redteam-framework-benchmark/hf_dataset.csv")

# Vulnerability rate by framework
print(df.groupby("framework")["panel_vulnerable"].mean())

# Decode panel votes
df["panel_votes_parsed"] = df["panel_votes"].apply(json.loads)

Reproduction

This dataset was produced by ORQ Research using three red-teaming frameworks
(EvaluatorQ, DeepTeam,
PromptFoo) against three victim LLMs.

Each row is a single attack attempt scored by a 3-model panel judge.

Limitations

• Single final turn only: attack_prompt and response are the last user turn and
  model response. For multi-turn attacks, intermediate turns are not included. The
  n_user_turns column shows how many turns the attack took.
• No judge reasoning: panel_votes stores only the boolean verdict per judge.
  The LLM judge's explanation is not persisted in this release.
• Benchmark scope: All attacks target API-accessible LLMs in three fixed
  configurations (vulnerable-agent, direct-model, secure-agent). Results may
  not generalise to production deployments with different system prompts or tooling.

Citation

@misc{orq2026redteambenchmark,
  title   = {ORQ Red-Teaming Framework Benchmark},
  author  = {ORQ Research Team},
  year    = {2026},
  url     = {https://huggingface.co/datasets/orq/redteam-framework-benchmark}
}

Blog post: Benchmarking LLM Red-Teaming Frameworks