license: cc-by-4.0
language:
- en
tags: - ai-safety
- agent-safety
- evals
- prompt-injection
- red-teaming
- security
pretty_name: Agentic Execution Guardrail Eval
Agentic Execution Guardrail Eval
Agentic Execution Guardrail Eval is a lightweight evaluation dataset for testing safety guardrails in agentic execution environments.
The dataset focuses on risky patterns that may appear when AI agents generate prompts, shell commands, or code intended for execution.
Scope
The evaluation cases cover:
- Shell execution and exfiltration patterns
- Hidden instructions
- Prompt injection attempts
- Jailbreak-style reframing
- Risky Python code patterns
- Dynamic imports and execution
- Agent safety edge cases
Intended Use
This dataset is intended for:
- AI safety research
- Agent evaluation
- Prompt injection testing
- Guardrail benchmarking
- Red teaming experiments
Architecture Assumption
Some records include an expected_layer field.
This field reflects the layered design of the original Agentic Execution Guardrail prototype, such as static signatures, AST analysis, learned jailbreak detection, or semantic policy checks.
It is optional metadata, not a required taxonomy. Other guardrail architectures may ignore this field and evaluate only the payload, expected verdict, category, and severity.
Records without this field are evaluated on payload, verdict, category, and severity alone.
Limitations
This dataset is not a comprehensive security benchmark.
The examples are manually curated and designed for lightweight guardrail evaluation rather than real-world security certification.
Citation
If you use this dataset in research, benchmarks, or evaluation pipelines, attribution is appreciated.
Author: Aleksei Khvostov