configs:
- config_name: default
default: true
data_files:- split: train
path: train.jsonl - split: test
path: test.jsonl
- split: train
- config_name: trim-10
data_files:- split: train
path: train-trim-10.jsonl - split: test
path: test-trim-10.jsonl
license: mit
task_categories:
- split: train
- text-classification
- zero-shot-classification
language: - en
tags: - safety
- shell
- bash
- classifer
- tool-call
size_categories: - 100K<n<1M
source_datasets: - openbmb/UltraData-SFT-Agent-2609
pretty_name: Tool Safety v2
Tool-safety classifier dataset
Source transcripts: openbmb/UltraData-SFT-Agent-2609
From conversations in the source datasets, tool calls are then passed through various LLMs to rank its safety given the conversation's context.
The contexts are synthetically generated if they cannot be inferred from the first system prompt or user prompt.
Configs:
- default: The full conversion. Picking from 2 tool calls per conversation from the source dataset
- trim-10:
promptstool calls are trimmed to last 10 tool calls after a user message
Columns:
| column | description |
|---|---|
context |
working_dir, git_repo, git_branch (best effort) |
prompts |
chronological single-key objects: user messages and already-executed tool calls |
tool_call |
single-key object {name: arguments} — bare value or key=value pairs (salient args for known tools, all args otherwise); raw arguments JSON only when unparsable/empty |
safety |
allow / ask / deny |
Label distribution
| safety | count |
|---|---|
| allow | 138424 |
| ask | 69212 |
| deny | 69212 |
Synthetic data provenance
ask/deny synthetic rows were generated deterministically by stage 3; deny commands partly derive from these external sources:
redcanaryco/atomic-red-team— MIT License, © Red Canary, Inc.zyw-286/shell-attack-evolution-dataset(Cowrie honeypot attacker commands) — CC BY 4.0
Policy
Labels follow the policy in policy.md