license: mit
task_categories:

• text-generation
  language:
• en
  tags:
• agent-security
• prompt-injection
• red-teaming
• safety
• nemo-gym
  pretty_name: ASB Selectors (materialized)
  size_categories:
• 10K<n<100K
  configs:
• config_name: default
  data_files:
  • split: validation
    path: all.jsonl

ASB Selectors — materialized evaluation rows

The Agent Security Bench (ASB) evaluation matrix, expanded once into concrete rows and
pinned, so that repeated runs score the identical inputs.

ASB ships construction tools rather than a dataset: the benchmark only exists once a
runner crosses agents × tasks × attacker tools and assembles the injected prompts. Two
people running upstream a week apart can therefore evaluate against different inputs. This
repo is one expansion of that matrix, frozen.

This is not an official ASB release. It is a derived artifact, published under ASB's
own MIT license (see LICENSE). For the benchmark itself, go upstream.

Why this exists

So that reproducing an ASB number does not require re-deriving the dataset first. You have
two options and they give you the same 10,800 rows:

• Use this frozen copy. Nothing to configure, no credentials, and everyone scoring
  against it is scoring identical inputs — which is what makes numbers comparable week to
  week and between teams.
• Regenerate it yourself from upstream's construction tools. The expansion is
  deterministic, so it produces these same rows; it just costs a clone and a few seconds,
  and it depends on GitHub being up and unchanged.

Neither option asks you to trust this repo: prepare verify in the NeMo Gym adapter
re-derives the rows from upstream and diffs the content hash, so you can check this copy
rather than take it on faith.

Contents

Rows are in OpenAI Responses API format: responses_create_params.input holds the
input messages, verifier_metadata holds the task, the attacker tool, the normal tools,
the injection method and the defense under test.

Denominators

Every published condition is 400 rows — 10 agents × 1 task × 40 agent-matched attacker
tools. PoT is 5 agents × 2 tasks × 40. 27 conditions × 400 = 10,800 rows.

That figure is recovered from the granularity of the published percentages rather than
assumed: defense-table cells quantize to 1/400, the main-table DPI/OPI/MP cells to 1/2000,
and Mixed only resolves on 1200 — three attack types, not five. main_attacker.py defaults
task_num to 1 and the DPI/OPI/MP/mixed driver never overrides it.

Provenance

• Upstream: agiresearch/ASB @ 1f561dccf92d55302368fa67679b4ba9d9c8fdc4
• Paper: Zhang, Yu, Bao, Yang et al. Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents. ICLR 2025. arXiv:2410.02644
• Expanded 2026-09-20. manifest.json carries the SHA-256 of every upstream file the
  expansion read, so drift is detectable rather than silent.

One upstream file, agent_task_pot_all.jsonl, is an HTML error page rather than JSONL in
this revision. It is recorded as invalid and never parsed.

Content warning

This dataset is adversarial by construction. It contains prompt-injection payloads and
attacker tool definitions describing actions such as unauthorized fund transfers and data
exfiltration. They are benchmark fixtures for measuring whether an agent can be diverted;
no tool here executes anything. Published for security evaluation and research.

Use

from huggingface_hub import snapshot_download

snapshot_download(repo_id="theverifier/asb-selectors", repo_type="dataset",
                  local_dir="resources_servers/asb/data")

With the NeMo Gym adapter:

python -m benchmarks.asb.prepare pull     # restore these pinned rows
python -m benchmarks.asb.prepare verify   # re-expand from upstream, diff the content hash

Scoring an agent against these rows involves documented deviations from upstream's
protocol (output-token cap, memory ranking, plan-parse salvage, message merging). Any
reported number should travel with that disclosure — see benchmarks/asb/METRICS.md in
the adapter.

License

MIT, inherited from upstream ASB. LICENSE reproduces the notice in full:

Copyright (c) 2024 AGI Research

If you use this, cite the ASB paper.