license: cc-by-4.0
language:
- en
size_categories: - n<1K
task_categories: - text-classification
tags: - agents
- agent-safety
- evaluation
- over-refusal
- authorization
- action-boundary
- benchmark
configs: - config_name: scenarios
data_files:- split: test
path: scenarios.jsonl
- split: test
- config_name: verdicts
data_files:- split: test
path: verdicts.jsonl
- split: test
SteerBench-Work v2026-05
A benchmark for the moment before an agent acts.
An agent has read the context, chosen a tool action, and is about to change the
world: send a message, update a record, merge code, charge a card. SteerBench-Work
asks one question at that boundary: should the action proceed, or should the system
hold for review?
Both mistakes are scored. Acting on work it should have held, and holding work it
was cleared to do.
The finding
Across 30 model conditions on 106 scenarios, five trials each:
| Error direction | What the model did wrong | Rate |
|---|---|---|
| Over-refusal | Held work it was authorized and cleared to do | 28.1% (471 / 1,677) |
| Under-refusal | Performed work it was not authorized to do | 1.0% (15 / 1,500) |
The two rates rest on near-identical denominators, 1,677 chances against 1,500, so
the gap is not a matter of more cases pointing one way.
On evidence-reversed mirrors of public incidents, accuracy falls to 63.8%
against 98.5% on the incidents themselves. We report that as a difficulty
result, not a cause.
What is in this dataset
| Config | Rows | Contents |
|---|---|---|
scenarios |
106 | The locked scenario set: context, evidence, decision point, expected action |
verdicts |
15,900 | Every model trial: the decision, policy action, confidence, and the model's own stated reason |
from datasets import load_dataset
scenarios = load_dataset("agentdock/steerbench-work-2026-05", "scenarios")
verdicts = load_dataset("agentdock/steerbench-work-2026-05", "verdicts")
The 106 original .json files are also included under scenarios/, byte-identical
to the locked release, so the published checksums verify against them. The .jsonl
files are a convenience rendering of the same data.
38 of the 15,900 trials returned no parseable reason. Those rows are kept withreason: null rather than dropped, so the counts stay exact.
Running it yourself
This dataset is the questions. The runner, the scorer, the validator, and the
release manifests are in the code repository:
https://github.com/AgentDock/steerbench-work
Two different things are possible there, and only one costs money:
- Check our results. Re-run the scorer over the published verdicts and verify
the checksums. No API key, no model calls, deterministic. - Evaluate a new model. Run the 106 scenarios against any model with your own
API key. The roster was frozen on June 8, 2026, so anything released after
that date is outside this release. Later evaluations appear in a separately
versioned release; this one is never retroactively rewritten.
Leaderboard and per-scenario verdicts: https://steerbench.com
Scenario texts and named companies
Scenario texts are constructed. References to named companies and incidents
point to public records such as court rulings, regulator actions, and press
coverage, and carry no claim about any organization's current systems. Nothing
in this release supports a claim about the safety of any deployed agent system.
Method, briefly
- Deterministic scoring. No model judges another model.
- One system prompt, pinned by SHA-256.
- Five independent trials per scenario and model condition. Mean trial accuracy,
the modal-of-5 verdict, and pass^5 are reported side by side, because models
flip between trials. - The answer key was hash-committed before any model ran.
- The scoring key is the benchmark-owner label set. A separate three-rater human
pass is reported as independent corroboration, not as scoring authority.
Limitations
The scenarios are constructed, single-turn descriptions of commit moments. A model
reads the moment; it does not execute tools. The format scores one boundary per
scenario, not a full session, and not what an agent does after a denial. Model
scores are snapshots of dated provider endpoints. The proceed-labeled and
hold-labeled pools are matched on opportunity count but not on authored difficulty.
Contamination
This is a public reference set. Publishing it starts a contamination clock: from
this date the scenarios and the answer key can enter training corpora. That is
deliberate and documented. Graded rankings move to fresh versioned sealed releases
as contamination accrues.
Licensing
The scenario sets and the results bundles, including the per-scenario verdicts,
are CC BY 4.0, Copyright (c) 2026 AgentDock. See LICENSE-DATA in the code
repository. The runner source code is MIT, licensed separately.
The reason field in verdicts contains text generated by third-party models
from OpenAI, Anthropic, Google, DeepSeek, Moonshot, and open-weight providers.
Downstream use of that text is additionally subject to the terms of the provider
that produced it.
Citation
Paper: SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries (arXiv:2608.12654).
@misc{serdar2026steerbenchwork,
title = {SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries},
author = {Serdar, Oguz and Mertayak, Cuneyt},
year = {2026},
eprint = {2608.12654},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2608.12654}
}
See CITATION.cff in the code repository for the machine-readable attribution.