evaluate 0 0 0

usezn/zn-prompt-injection-bench

published by @usezn

Alignment dataset tracked in the /datasets sub-catalog.

Lifetime downloads
361
Last 30 days
146
Likes
2
Size
47.3 MB
Created on HF
2026-08-24
Age
1 months ago

Description

ZN Prompt-Injection Evaluation Corpus (v15)

A 23,699-row binary text-classification corpus for detecting prompt injection,

jailbreak attempts, and indirect attacks against LLM-based systems, paired with

hard adversarial negatives drawn from real human text. Every row carries its

source, license, language, and full provenance.

🛡️ Real-Time Defense for Agents: Evaluating models or securing your AI workflow? Check out zn-gate on npm (npx -y zn-gate mcp) and the open-source… See the full description on the dataset page: https://huggingface.co/datasets/usezn/zn-prompt-injection-bench.

Tags

task_categories:text-classificationlanguage:enlanguage:eslanguage:frlanguage:delanguage:ptlanguage:itlanguage:nllanguage:pllanguage:rulicense:cc-by-4.0size_categories:10K<n<100Kformat:parquetmodality:textlibrary:datasetslibrary:pandaslibrary:polarslibrary:mlcroissantregion:usprompt-injectionllm-securitysecurityadversarialai-safety

README current version from Hugging Face


license: cc-by-4.0
language:

  • en
  • es
  • fr
  • de
  • pt
  • it
  • nl
  • pl
  • ru
    size_categories:
  • 10K<n<100K
    task_categories:
  • text-classification
    tags:
  • prompt-injection
  • llm-security
  • security
  • adversarial
  • ai-safety
    pretty_name: ZN Prompt-Injection Evaluation Corpus v15

zn — prompt-injection gate for AI agents

ZN Prompt-Injection Evaluation Corpus (v15)

A 23,699-row binary text-classification corpus for detecting prompt injection,
jailbreak attempts, and indirect attacks against LLM-based systems, paired with
hard adversarial negatives drawn from real human text. Every row carries its
source, license, language, and full provenance.

🛡️ Protect your coding agent: zn-gate is a local security layer for coding agents (Claude Code, Codex CLI, Cursor, OpenCode). It blocks prompt injection and keeps your secrets away from the model. Set it up with npx -y zn-gate init (install guide). No account, no telemetry. Source on GitHub.

  • Rows: 23,699 (train 16,888 · validation 3,301 · test 3,510)
  • Size: about 49.5 MB in total (47.2 MiB) for all data files. data/corpus_v15.csv is 26.7 MB and data/pool_v15.parquet is 11.2 MB; both hold all 23,699 rows. The three split files (train, validation, test) total 11.6 MB and hold the same rows, divided by split. Download only the files you need.
  • Labels: ataque (attack) 12,317 · benigno (benign) 11,382
  • Languages: English plus 8 translation locales (es, fr, de, pt, it, nl, pl, ru); short benign prompts add small amounts of roughly 40 additional locales.
  • Corpus-level license: CC-BY-4.0, with per-row upstream licenses preserved in the license column (see Licensing).

The label tokens follow the project's original vocabulary: ataque means the
text contains a prompt-injection, jailbreak, or manipulation attempt;
benigno means ordinary non-adversarial text.

Why this corpus was rebuilt

An earlier iteration (7,241 fully synthetic rows) failed evaluation: the model
memorized a synthetic template manifold, so real out-of-distribution text
scored as an attack while genuine plain-language attacks slipped below the
benign manifold. v15 rebuilds the pool exclusively from independently licensed
sources, removes the legacy synthetic corpus, adds real long-form,
conversational, and security-meta-textual benign coverage, and extends the
attack side with deterministic paraphrases and machine-translated
multilingual attacks under a leak-audited group-aware split.

Composition

By family, label, and split (counts generated directly from the released files):

Family Label Train Validation Test Total Share
plain ataque 3,021 613 631 4,265 18.0%
red_team ataque 2,719 586 730 4,035 17.0%
paraphrase ataque 1,822 177 206 2,205 9.3%
multilingual_attack ataque 1,358 216 238 1,812 7.6%
corto benigno 2,703 579 578 3,860 16.3%
largo_real benigno 2,266 486 486 3,238 13.7%
conversacional benigno 1,692 363 362 2,417 10.2%
meta_texto benigno 677 145 144 966 4.1%
documentacion benigno 630 136 135 901 3.8%
Total 16,888 3,301 3,510 23,699 100%

Family glossary: plain = direct natural-language injection/jailbreak prompts;
red_team = multi-turn manipulation and indirect injection through tool or
document context; paraphrase = mechanically derived rewordings of attack
seeds; multilingual_attack = machine-translated attack seeds; corto =
short real user-style requests; largo_real = continuous long-form prose;
conversacional = multi-turn human-assistant dialogue; meta_texto =
security-themed text about attacks without live payloads; documentacion =
technical documentation excerpts.

Language coverage:

Language(s) Rows Where
en 10,996 Verified-English rows across the five benign families
(unmarked, English-derived) 10,505 Attack families plain, red_team, paraphrase
es, fr, de, pt, it, nl, pl, ru 1,812 multilingual_attack translations (fr 228, es 227, pt 227, it 227, nl 227, pl 227, de 225, ru 224)
~40 additional locales 386 Short benign prompts from MASSIVE, native locale marked

No single multilingual locale exceeds 12.6% of the multilingual bucket.

Split protocol

Splits were produced by a group-aware MinHash near-duplicate pipeline
(seed 42):

  1. Texts are shingled into character 8-grams and embedded into 128-permutation
    MinHash signatures.
  2. Locality-sensitive hashing (32 bands x 4 rows) with a Jaccard threshold of
    0.5 groups exact and near-duplicate texts into groups.
  3. Groups - not individual rows - are assigned to train / validation /
    test at 70/15/15, so near-duplicates can never straddle splits.
  4. Each machine translation is force-paired with its seed before assignment
    (cross-language pairs cannot be linked by n-gram similarity), guaranteeing
    co-splitting of seed and translation.
  5. Post-assignment audit: 0 violations (no exact or near-duplicate pair
    crosses a split), split ratios within +/-10 percentage points of target.

Data fields

Column Type Description
text string Document or prompt content
label string ataque (attack) or benigno (benign)
family string One of the nine families listed above
source string Source identifier (see Licensing)
license string Upstream license for the row's origin material
lang string Language tag; empty for the English-derived attack families
char_len int Character length of text
provenance string Row lineage: source item reference; for derived rows, the seed identifier plus transform details; for translated rows, seed id, locale, and model
split string Present only in pool_v15.parquet and corpus_v15.csv

Files: data/train.parquet, data/validation.parquet,
data/test.parquet (standard split naming), plus the complete pool as
data/pool_v15.parquet and data/corpus_v15.csv.

Provenance and licensing

Every row retains its upstream license. Sources, verified against their
official repositories or dataset metadata, and row counts:

Source Origin License Rows
hh_rlhf Anthropic hh-rlhf MIT 6,437
injecagent InjecAgent benchmark MIT 2,944
aegis_2.0 NVIDIA Aegis AI Content Safety 2.0 CC-BY-4.0 2,541
dolly_15k Databricks Dolly-15k CC-BY-SA-3.0 2,209
wikipedia_en Wikimedia Wikipedia (en) CC-BY-SA-3.0+GFDL 1,551
c4_en C4 (domain-filtered) ODC-BY 1,541
python_docs Python official documentation PSF 1,529
gsm8k OpenAI GSM8K MIT 1,354
advbench AdvBench behaviors MIT 742
oasst1 OpenAssistant oasst1 Apache-2.0 872
prompt_injections deepset prompt-injections Apache-2.0 394
massive Amazon MASSIVE CC-BY-4.0 390
rfc_editor RFC Editor IETF-Trust-Legal-Provisions 273
jbb_behaviors JailbreakBench JBB-Behaviors MIT 154
helpsteer2 NVIDIA HelpSteer2 CC-BY-4.0 479
gutenberg Project Gutenberg Public-domain-US 146
agentdojo ETH Zurich AgentDojo MIT 112
bipia Microsoft BIPIA MIT 31

Notes:

  • The corpus-level license is CC-BY-4.0. Where an upstream license carries
    attribution or share-alike terms (for example CC-BY-SA on Dolly and
    Wikipedia, ODC-BY on C4), those upstream terms continue to govern the
    underlying text.
  • The rfc_editor contributions reproduce text under the
    IETF-Trust-Legal-Provisions license, which permits reproduction with
    attribution. gutenberg rows are US public-domain literature
    (Public-domain-US); wikipedia rows carry the dual
    CC-BY-SA-3.0+GFDL grant.
  • Restriction notice: an additional 1,416 rows derived from a proprietary
    internal mutation set (internal license, redistribution not granted)
    are not included in this release; they are reserved privately and do not
    appear in any released file.
  • No source with unknown or non-permissive licensing was used. Sources
    rejected during clearance (non-commercial or unlicensed user-generated
    dumps) contribute zero rows.

Multilingual translation methodology

The multilingual_attack family contains 1,812 machine-translated attack
texts in eight locales (es, fr, de, pt, it, nl, pl, ru):

  • Seeds were selected deterministically by stable hash from existing
    labeled attacks in the pool (no ad-hoc picking, reproducible selection).
  • Translation used the Qwen3-32B model served through Amazon Bedrock
    (qwen.qwen3-32b-v1:0) with temperature 0.1 and extended thinking
    disabled.
  • Model responses were frozen to disk per seed before any downstream use, so
    rebuilding the dataset replays identical payloads instead of re-invoking
    the model.
  • Automated validation kept only strict outputs: a complete JSON envelope
    covering all eight locales, output differing from its seed, uniqueness
    across locales, and script-direction checks (Cyrillic-majority for Russian,
    Latin-script elsewhere).
  • Total translation spend was USD 0.07 for the full bucket, recorded
    per-call in a cost ledger.

These translations are machine-generated: quality is medium, occasional
mechanical errors and unnatural phrasing remain, and no professional human
translation review has been performed. Treat the multilingual bucket as a
cross-lingual robustness surface, not as gold-standard natural text.

Limitations

  • Translation quality is medium. Machine-generated translations may
    contain errors; sample and review them before high-stakes use.
  • Declared composition gaps. Relative to the planned composition, three
    families finished under target: plain at 61% (4,265 of 7,000),
    documentacion at 45% (901 of 2,000), and long-form prose at 81%
    (3,238 of 4,000). The gaps come from gated upstream sources and finite
    document pools; they were left open rather than silently patched.
  • Paraphrases are mechanical. The paraphrase family uses a fixed
    synonym table, safe clause reordering, and fixed contextual prefixes, so it
    anchors surface-form variation but does not provide deep semantic diversity.
  • Legacy evaluation overlap. 135 rows have exact twins in an older
    evaluation subset; if that subset is reused for external evaluation,
    exclude its twins from one side first.
  • Adversarial content. The corpus intentionally contains jailbreaks,
    injection payloads, and manipulative text. Use it for defensive research
    and evaluation of detection systems under applicable policies.
  • An additional 1,416 held-out rows (synthetic mutants derived from our
    internal production corpus) are reserved privately and are not part of this
    public release.

Citation

@dataset{zn_prompt_injection_corpus_v15,
  title     = {ZN Prompt-Injection Evaluation Corpus (v15)},
  author    = {zn},
  year      = {2026},
  version   = {v15},
  url       = {https://github.com/usezn/zn}
}

README history 6 revisions

Every night we snapshot the README of every dataset in the catalog. When the SHA changes we archive the new version and diff it against the last. This is the evolving thought record of the alignment-data field: what the author decided to say about the corpus, and how that framing shifted over time.

  1. 2026-10-20Update the zn-gate box: current agents and install command02dbca611.6 KB
    Loading...
  2. 2026-10-14Card: real dataset size and English wording of the internal licence notee08465311.5 KB
    Loading...
  3. 2026-10-10docs: point links to usezn org (dataset + GitHub repo moved); author zn4ed992a11.3 KB
    Loading...
  4. 2026-09-06docs: add brandkit banner and zn-gate agent security links6ae877511.4 KB
    Loading...
  5. 2026-08-24Release zn.dataset v15: prompt-injection eval corpus (23,699 rows)eafc6c210.5 KB
    Loading...
  6. 2026-08-24initial commita10199d0 B
    Loading...

Used by abliterated models none yet

No abliterated models in our catalog have declared this dataset in their YAML metadata yet. This can mean the dataset is used but not declared, is used outside abliteration workflows, or is new. The link will populate automatically as models are indexed.

Community discussions 1 thread · 1 comments

Threads posted by the community on Hugging Face: bug reports, questions about usage, pull requests on the README, license clarifications. Click any thread to expand and read the full conversation inline. Newest first.

  1. 2026-08-25[bot] Conversion to Parquetclosed1 💬#1
    Loading...

Related datasets same stage · evaluate

Read further

Metadata

License
cc-by-4.0

Download the dataset

0

Copy any of these snippets into your notebook or terminal. All three fetch directly from Hugging Face using your own credentials.

Recommended: datasets library (Python)
from datasets import load_dataset
ds = load_dataset("usezn/zn-prompt-injection-bench")
Raw snapshot (Python)
from huggingface_hub import snapshot_download
snapshot_download(repo_id="usezn/zn-prompt-injection-bench", repo_type="dataset")
git clone (requires git-lfs)
git clone https://huggingface.co/datasets/usezn/zn-prompt-injection-bench
Catalog is the map. Apps are the tools.

Run models on your own machine, not in the cloud.

Every model page has an "Open in Abliteration" button that hands the model directly to the first-party desktop client, at the quantization your rig can actually run. No API keys, no subscription, no prompt leakage.

Open in Abliteration