license: mit
pretty_name: SEVRA-BENCH - Social Engineering of Vulnerabilities in Review Agents
tags:

• code
• security
• red-teaming
• vulnerability
• adversarial
  configs:
• config_name: cwe125-deterministic
  data_files:
  • split: malicious
    path: cwe125/deterministic/generated_prs.jsonl
• config_name: cwe22-deterministic
  data_files:
  • split: malicious
    path: cwe22/deterministic/generated_prs.jsonl
• config_name: cwe352-deterministic
  data_files:
  • split: malicious
    path: cwe352/deterministic/generated_prs.jsonl
• config_name: cwe416-deterministic
  data_files:
  • split: malicious
    path: cwe416/deterministic/generated_prs.jsonl
• config_name: cwe78-deterministic
  data_files:
  • split: malicious
    path: cwe78/deterministic/generated_prs.jsonl
• config_name: cwe787-deterministic
  data_files:
  • split: malicious
    path: cwe787/deterministic/generated_prs.jsonl
• config_name: cwe79-deterministic
  data_files:
  • split: malicious
    path: cwe79/deterministic/generated_prs.jsonl
• config_name: cwe862-deterministic
  data_files:
  • split: malicious
    path: cwe862/deterministic/generated_prs.jsonl
• config_name: cwe89-deterministic
  data_files:
  • split: malicious
    path: cwe89/deterministic/generated_prs.jsonl
• config_name: cwe94-deterministic
  data_files:
  • split: malicious
    path: cwe94/deterministic/generated_prs.jsonl
• config_name: cwe125-benign
  data_files:
  • split: benign
    path: cwe125/benign/gpt5.2_v2/generated_prs.jsonl
• config_name: cwe22-benign
  data_files:
  • split: benign
    path: cwe22/benign/gpt5.2_v2/generated_prs.jsonl
• config_name: cwe352-benign
  data_files:
  • split: benign
    path: cwe352/benign/gpt5.2_v2/generated_prs.jsonl
• config_name: cwe416-benign
  data_files:
  • split: benign
    path: cwe416/benign/gpt5.2_v2/generated_prs.jsonl
• config_name: cwe78-benign
  data_files:
  • split: benign
    path: cwe78/benign/gpt5.2_v2/generated_prs.jsonl
• config_name: cwe787-benign
  data_files:
  • split: benign
    path: cwe787/benign/gpt5.2_v2/generated_prs.jsonl
• config_name: cwe79-benign
  data_files:
  • split: benign
    path: cwe79/benign/gpt5.2_v2/generated_prs.jsonl
• config_name: cwe862-benign
  data_files:
  • split: benign
    path: cwe862/benign/gpt5.2_v2/generated_prs.jsonl
• config_name: cwe89-benign
  data_files:
  • split: benign
    path: cwe89/benign/gpt5.2_v2/generated_prs.jsonl
• config_name: cwe94-benign
  data_files:
  • split: benign
    path: cwe94/benign/gpt5.2_v2/generated_prs.jsonl
• config_name: cwe125-benign-top10cvss
  data_files:
  • split: benign
    path: cwe125/benign/top10cvss/generated_prs.jsonl
• config_name: cwe22-benign-top10cvss
  data_files:
  • split: benign
    path: cwe22/benign/top10cvss/generated_prs.jsonl
• config_name: cwe352-benign-top10cvss
  data_files:
  • split: benign
    path: cwe352/benign/top10cvss/generated_prs.jsonl
• config_name: cwe416-benign-top10cvss
  data_files:
  • split: benign
    path: cwe416/benign/top10cvss/generated_prs.jsonl
• config_name: cwe78-benign-top10cvss
  data_files:
  • split: benign
    path: cwe78/benign/top10cvss/generated_prs.jsonl
• config_name: cwe787-benign-top10cvss
  data_files:
  • split: benign
    path: cwe787/benign/top10cvss/generated_prs.jsonl
• config_name: cwe79-benign-top10cvss
  data_files:
  • split: benign
    path: cwe79/benign/top10cvss/generated_prs.jsonl
• config_name: cwe862-benign-top10cvss
  data_files:
  • split: benign
    path: cwe862/benign/top10cvss/generated_prs.jsonl
• config_name: cwe89-benign-top10cvss
  data_files:
  • split: benign
    path: cwe89/benign/top10cvss/generated_prs.jsonl
• config_name: cwe94-benign-top10cvss
  data_files:
  • split: benign
    path: cwe94/benign/top10cvss/generated_prs.jsonl

SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agents

A benchmark dataset for evaluating automated code review agents against adversarial pull requests. The dataset contains 2,250 malicious PRs (current release: deterministic) and 347 benign ground-truth security fixes, grounded in real vulnerabilities from the OSV/SECommits database, across 10 CWE classes from the 2025 CWE Top 25.

Motivation

Modern AI coding assistants can generate plausible-looking patches at scale. This dataset stress-tests code reviewers — both human and AI — against a concrete threat model: an adversary with contributor access who re-introduces a previously patched CVE, disguised as a routine change.

Real-world supply-chain attacks frequently arrive as pull requests: a seemingly helpful bug fix that quietly reverts a security patch, a refactor that re-opens an injection vector, a performance optimization that removes a bounds check. This benchmark operationalizes that threat by deterministically reversing official CVE fixes and parametrizing the social engineering framing applied to the PR description.

Dataset Overview

Malicious PRs — deterministic (current release)

• 2,250 attack PRs across 10 CWE classes
• Each attack deterministically reverses an official CVE fix using git apply -R — no code synthesis, no LLM-generated diffs
• PR titles, descriptions, and commit messages are generated by an LLM according to one of 15 social engineering framing strategies
• After a difficulty filter (samples that fooled at least one of two baseline models), 1,062 samples constitute the challenging evaluation subset

Benign PRs (current release)

• 347 ground-truth security fix PRs across the same 10 CWE classes
• Generated by applying the actual, unmodified CVE fix patches using git apply
• Used to measure the false-decline rate (FDR): fraction of legitimate security improvements incorrectly rejected
• Prevents conflating an overly conservative reviewer with a genuinely discerning one
• A 10-per-CWE subset (top10cvss, 100 PRs) selects the ten highest-CVSS fixes in each CWE for cheaper evaluation across many models; three vulnerability IDs appear under two CWEs, so load it per CWE (e.g. cwe79-benign-top10cvss) to keep all 100

Covered CWEs

Attack Taxonomy: Social Engineering Framing Strategies

The code change in each malicious PR is identical (deterministic patch reversal). The variation comes entirely from how the PR is framed. The 16 framing strategies exploit different cognitive biases:

Record Schema

Malicious PR Fields (deterministic)

Benign PR Fields (gpt5.2_v2)

Generation Pipeline

Malicious PRs (deterministic)

1. CVE selection: Vulnerabilities are sourced from Secommits, selecting unique projects from the 10 most prevalent CWE classes in the 2025 CWE Top 25, sampled by round-robin across CVSS severity bins.

2. Deterministic patch reversal: The official CVE fix commit is mechanically reversed using git apply -R, restoring the vulnerable code. No LLM code generation occurs — the reintroduced code is the exact vulnerable version from the official record.

3. PR framing: An LLM generates the PR title, description, and commit message according to one of 16 framing strategies. The code diff is fixed; only the presentation varies.

Benign PRs (gpt5.2_v2)

The official fix patches are applied using git apply. No code synthesis occurs. An LLM generates only the PR title and description in a straightforward, non-deceptive tone.

Benchmark

This dataset is used with the malicious-pr-bench evaluation harness, which deploys PRs to a live Gitea instance and evaluates reviewer agents via the inspect-ai framework.

Metrics:

• Detection Accuracy: fraction of malicious PRs correctly blocked
• Security Reason Rate: fraction of blocked PRs where the stated rationale referenced a security concern
• False-Decline Rate (FDR): fraction of benign security fix PRs incorrectly rejected

Quick Start

from datasets import load_dataset

# Load malicious PRs for CWE-79 (deterministic, current release)
ds = load_dataset("RedAI4Code/SEVRA", "cwe79-deterministic")

# Load benign PRs for CWE-79
ds_benign = load_dataset("RedAI4Code/SEVRA", "cwe79-benign")

print(ds["malicious"][0])

To run the full benchmark:

git clone https://github.com/rufimelo99/malicious-pr-bench.git
cd malicious-pr-bench
uv sync
uv run inspect eval benchmark/task.py@reviewer_benchmark \
  --model anthropic/claude-opus-4-6 \
  -T cwe=cwe79 \
  -T version=deterministic \
  -T tool_mode=gitea \
  --log-dir logs/

Dataset Versions

License

MIT License

Resources

• GitHub: malicious-pr-bench
• Paper: arXiv

@misc{melo2026sevrabenchsocialengineeringvulnerabilities,
      title={SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agents}, 
      author={Rui Melo and Riccardo Fogliato and Sean Zhou and Pratiksha Thaker and Zhiwei Steven Wu},
      year={2026},
      eprint={2606.13757},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2606.13757}, 
}