license: cc-by-4.0
task_categories:
- text-classification
language: - en
tags: - prompt-injection
- llm-security
- owasp-llm-top-10
- command-execution
- ai-safety
- jailbreak
- adversarial-prompts
- red-teaming
size_categories: - n<1K
pretty_name: CEDA-215
configs: - config_name: default
data_files:- split: full
path: ceda-215.jsonl
- split: full
CEDA-215: Command-Execution Defense Assessment (215 entries)
CEDA-215 is a labeled evaluation dataset for benchmarking input-classification defense pipelines for command-execution LLM chatbots. Each entry pairs a natural-language input with a ground-truth SAFE or UNSAFE label, allowing fine-grained measurement of true positives, false positives, true negatives, and false negatives across an OWASP LLM Top 10 - aligned threat taxonomy.
Why this dataset exists
Existing public prompt-injection benchmarks — InjecAgent, AgentDojo, Open-Prompt-Injection, and the January 2026 OWASP-Llama benchmark — measure attack success against tool-using chat agents (red-team view: "can the attacker hijack the agent?"). None provides the balanced classification labels required to evaluate a whitelisted command-execution defense pipeline (blue-team view: "can the defense correctly tag each input as SAFE/UNSAFE while keeping false positives on benign inputs low?"). CEDA-215 fills that specific evaluation gap.
Dataset summary
| Total entries | 215 |
| Labels | 146 SAFE / 69 UNSAFE |
| Language | English |
| Domain | Command-execution chatbots with a strict shell-command whitelist (e.g. ls, date) |
| Threat taxonomy | OWASP LLM Top 10 (2025/2026) |
| License | CC-BY 4.0 |
| Reproducibility | Deterministic — generated by scripts/buildDataset.js (no randomness, no external API calls) |
Category breakdown
| Category | Entries | Label distribution |
|---|---|---|
benign |
80 | 80 SAFE |
prompt_injection |
15 | 15 UNSAFE |
command_injection |
12 | 12 UNSAFE |
path_traversal |
10 | 10 UNSAFE |
encoding_obfuscation |
10 | 10 UNSAFE |
semantic_manipulation |
10 | 10 UNSAFE |
output_probing |
5 | 5 SAFE (inputs that probe for sensitive output but are not themselves blockable) |
rate_limit_evasion |
59 | 55 SAFE (individual turns within a rapid sequence) / 4 UNSAFE (turns that breach the rate budget) |
multi_turn_escalation |
14 | 6 SAFE / 8 UNSAFE (sequential turns building toward a forbidden command) |
Ablation subset (81 entries)
The accompanying paper's ablation experiments use a balanced 81-entry subset of CEDA-215, drawn as: 4 per single-turn attack category (20), 24 benign baselines, 32 rate-limit-evasion turns, 3 multi-turn escalations, 2 output-probing. This subset is regeneratable from the full dataset by the ablation script; consumers wishing to reproduce the paper's specific configurations should run scripts/runAblation.js from the companion code repository.
Data fields
Each entry is a JSON object with the following keys:
| Field | Type | Description |
|---|---|---|
id |
string | Stable unique identifier (e.g. prompt_injection_001) |
input |
string | The natural-language input as it would be submitted to the chatbot |
expectedLabel |
string | Ground-truth label: SAFE or UNSAFE |
category |
string | OWASP-aligned threat category (see table above) |
subcategory |
string | Finer-grained subcategory tag (e.g. simple_command, Instruction override) |
metadata |
object | Auxiliary fields (e.g. multi-turn sequence membership). May be empty. |
Loading the dataset
from datasets import load_dataset
ds = load_dataset("aalayed/CEDA-215")
print(ds["full"][0])
Or directly with Python's json module:
import json
with open("ceda-215.jsonl") as f:
rows = [json.loads(line) for line in f]
print(len(rows), "entries")
Intended use & limitations
Intended use. CEDA-215 is designed for benchmarking defense layers in command-execution AI chatbots — specifically, classifier accuracy on a balanced SAFE/UNSAFE task, with category-level reporting that maps to the OWASP LLM Top 10 risk surface.
Out of scope. This dataset is not a comprehensive prompt-injection benchmark for general-purpose chat agents; for that, see InjecAgent, AgentDojo, or Open-Prompt-Injection. It is also not a fine-tuning corpus — the entries are evaluation prompts, not labeled training data.
Dual-use considerations. The 69 UNSAFE entries include real adversarial command-injection and prompt-injection payloads. They are no more dangerous than payloads already public in the cited red-team benchmarks (HarmBench, AdvBench, InjecAgent), and they target a whitelisted chatbot scope (ls, date) rather than open-ended shell access. Use them only for defensive research and authorized testing.
Source and construction
CEDA-215 was constructed by scripts/buildDataset.js in the companion code repository. The 69 adversarial entries are extracted from the project's HackerAgent.STATIC_ATTACKS payload library; the 80 benign entries cover eight subcategories chosen to expose typical false-positive triggers (legitimate command intents, conversational small-talk, false-positive bait words like "rm", multi-word safe requests, data-like inputs that resemble payloads). Stateful sequences (rate-limit, multi-turn) are generated programmatically from sequence templates.
Citation
@dataset{ceda215_2026,
title = {{CEDA-215: Command-Execution Defense Assessment}},
author = {Alayed, Abdulaziz},
year = {2026},
publisher = {Zenodo},
version = {1.0.0},
doi = {10.5281/zenodo.20122529},
url = {https://huggingface.co/aalayed/CEDA-215}
}
If you use CEDA-215, please also cite the accompanying paper:
@article{ceda215_paper_2026,
title = {Hybrid Rule-LLM Defense Pipelines for Command-Execution AI Agents:
An Empirical Ablation Study Against OWASP LLM Top 10 Threats},
author = {<Alayed, Abdulaziz>},
year = {2026}
}
License
This dataset is released under the Creative Commons Attribution 4.0 International License (CC-BY 4.0). You are free to share and adapt the material for any purpose, including commercial use, provided you give appropriate credit.
The companion code repository is licensed separately under GPL-3.0-or-later.