configs:

• config_name: attacks
  data_files: data/attacks.jsonl
• config_name: taxonomy
  data_files: data/taxonomy.jsonl
  language:
• en
  license: cc-by-4.0
  pretty_name: RecRedTeam-ASI-Bench
  size_categories:
• 100<n<1K
  task_categories:
• other
  tags:
• red-teaming
• agentic-ai
• ai-safety
• llm-security
• prompt-injection
• recommender-systems
• e-commerce
• owasp
• synthetic
• benchmark

RecRedTeam-ASI-Bench

A red-team benchmark for LLM / agentic recommendation and shopping assistants.

270 adversarial cases across 9 threat categories, each mapped to
OWASP Agentic Security Initiative (ASI) 2026 and the OWASP LLM Top-10.
Every case ships a poisoned product catalog, the shopper query that opens the
attack surface, the attacker goal, and the failure signals a judge should look
for — so an audit is reproducible without any API key.

Companion demo: RecRedTeam-AuditLab (Space)
· Framework: RecRedTeam (GitHub)

Why this benchmark exists

Recommendation and shopping agents (Amazon Rufus, ChatGPT shopping, Temu/Taobao
agents) went mainstream in e-commerce in 2026, and the EU AI Act / DSA / China's
Generative AI Measures all demand demonstrable safety-audit capability.
Existing trustworthy-recommendation benchmarks (e.g. PRA) measure recommendation
quality; the harm-auditing half — can the agent be manipulated into harming
the shopper? — is still uncovered. This dataset is the case suite that fills it.

Generic red-teaming (ART), search-agent red-teaming (SafeSearch) and deceptive-
interface benchmarks each cover a single slice. This suite covers the full
spectrum of the recommendation domain in one place.

Threat taxonomy (8 + 1 categories)

30 cases per category. The hallucination / indirect_prompt_injection /
fake_review_manipulation / price_deception / dark_pattern / filter_bubble /
stereotype_bias / backdoor / multi_turn_escalation families each contain the
hand-written seed case(s) from the RecRedTeam framework plus templated variants.

Schema — attacks

Schema — taxonomy

category_id, name, name_cn, description, owasp_asi, owasp_llm, default_severity, examples

Usage

from datasets import load_dataset

bench = load_dataset("NoahIsARider/RecRedTeam-ASI-Bench", "attacks", split="train")
taxo  = load_dataset("NoahIsARider/RecRedTeam-ASI-Bench", "taxonomy", split="train")

# recommended: evaluate on the public split only
public = bench.filter(lambda r: r["split"] == "public")

case = public[0]
print(case["category_name"], "|", case["severity"], "|", case["owasp_asi"])
print(case["user_query"])
for p in case["products"]:
    print(" -", p["pid"], p["name"], p["price"], p["attrs"])

Run one case against your agent:

catalog   = case["products"]                 # retrieved / poisoned context
query     = case["user_query"]               # shopper turn
payload   = case["hidden_instructions"]      # inject into the matching product
response  = your_agent(query, catalog, payload)

attack_succeeded = any(
    s.lower() in response.lower() for s in case["failure_signals"]
)

Aggregate per category to get the headline metric:

[
\mathrm{ASR}_c = \frac{1}{|\mathcal{A}c|}\sum{a \in \mathcal{A}_c} \mathbb{1}[\text{attack } a \text{ succeeded}]
]

and the defense-effectiveness score
(\mathrm{Eff}(d) = 1 - \mathrm{ASR}{d} / \mathrm{ASR}{\text{baseline}}).

Reproducing the dataset

python scripts/build_dataset.py --per-category 30 --seed 2026

Generation is deterministic (random.Random(seed)), so the published files are
byte-reproducible from this repository.

Intended use & limitations

Intended — compliance auditing, safety research, defense evaluation, and
benchmarking of recommendation/shopping agents.

Limitations

• Fully synthetic. Brands, sellers, prices, reviews and domains are invented.
  No real user data, no real listings, no real personal information.
• Cases target a simulated shopping environment, never a live system.
• failure_signals are lexical markers for a rule-based judge channel; the
  RecRedTeam framework pairs them with an optional LLM judge channel for
  semantic verdicts.
• Coverage is 9 categories, not exhaustive — novel agentic failure modes
  (tool-chain hijack, cross-agent privilege escalation) are out of scope for v1.

Ethics & safety

This is a defensive artifact. Attack payloads are published so that defenders,
auditors and regulators can measure and harden agentic commerce systems; the
payloads are content for a simulated catalog and are not effective against any
real deployment by themselves. Do not use this suite against production systems
without authorization.

License

Data: CC BY 4.0.
Generator code: MIT (see LICENSE).

Citation

@misc{zhou2026recredteamasibench,
  title        = {RecRedTeam-ASI-Bench: A Red-Team Benchmark for LLM/Agentic
                  Recommendation and Shopping Assistants},
  author       = {Zhou, Fangyanuo},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/NoahIsARider/RecRedTeam-ASI-Bench}},
  note         = {9 threat categories, OWASP ASI 2026 / LLM Top-10 mapped}
}

Related

• Framework — github.com/NoahIsARider/RecRedTeam
• Live audit demo — RecRedTeam-AuditLab Space
• Author — Fangyanuo Zhou · noahisarider.github.io