license: mit
tags:

• cybersecurity
• llm-security
• benchmark
• red-teaming
• code-security
  pretty_name: CyberSecEval 3

CyberSecEval 3

This dataset packages the CyberSecEval 3 benchmark data published by Meta as part of the
PurpleLlama project, reformatted into a single
data.jsonl file with a category field identifying which CyberSecEval 3 sub-benchmark each
record belongs to.

Source

• Official repository: meta-llama/PurpleLlama
  (CybersecurityBenchmarks/datasets/)
• Project page: CyberSecEval 3 — Meta AI Research
• Retrieved: 2026-09-17, from the main branch of the repository (shallow clone).

Note: the meta-llama/PurpleLlama repository is a living monorepo that now hosts CyberSecEval 4
(the successor to CyberSecEval 3) in the same CybersecurityBenchmarks/datasets/ folder. This
package includes only the sub-benchmarks that were part of CyberSecEval 3; the two
CyberSecEval-4-only additions (autopatch/ — AutoPatchBench, and crwd_meta/ — CyberSOCEval,
built with CrowdStrike) were deliberately excluded. See "Known issues" below.

Paper

Wan, S., Nikolaidis, C., Song, D., Molnar, D., Crnkovich, J., Grace, J., Bhatt, M., Chennabasappa,
S., Whitman, S., Ding, S., Ionescu, V., Li, Y., & Saxe, J. (2024). CyberSecEval 3: Advancing the
Evaluation of Cybersecurity Risks and Capabilities in Large Language Models.
arXiv:2408.01605. https://arxiv.org/abs/2408.01605

@misc{wan2024cyberseceval3advancingevaluation,
      title={CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models},
      author={Shengye Wan and Cyrus Nikolaidis and Daniel Song and David Molnar and James Crnkovich and Jayson Grace and Manish Bhatt and Sahana Chennabasappa and Spencer Whitman and Stephanie Ding and Vlad Ionescu and Yue Li and Joshua Saxe},
      year={2024},
      eprint={2408.01605},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2408.01605}
}

License

MIT License — Copyright (c) Meta Platforms, Inc. and affiliates, as stated in
CybersecurityBenchmarks/LICENSE
of the source repository. This applies to the benchmark code and datasets; it is distinct from the
LLAMA Community License that covers the Llama Guard model weights hosted elsewhere in the same
monorepo. Redistribute in accordance with the MIT License terms (retain the copyright notice).

File

• data.jsonl — 9,361 records, one JSON object per line, UTF-8, each with a category field.

Categories included

All original fields from each source file are preserved unchanged; only a category field was
added (via Add-Member, forced) to every record before concatenation into JSONL.

Known issues / decisions made while preparing this package

1. autopatch/ and crwd_meta/ excluded. Per the source repository's own README, these two
   folders (AutoPatchBench and CyberSOCEval/CrowdStrike malware-analysis & threat-intel-reasoning)
   were introduced in CyberSecEval 4, not CyberSecEval 3, even though they now live in the same
   datasets/ directory. Including them would misrepresent this package as CyberSecEval 3.
2. instruct.json (v1) excluded in favor of instruct-v2.json. Both files exist in the
   repository with near-identical schemas; v2 is the revised/superseding version, so only v2 was
   used to avoid near-duplicate records. If you need v1 as well it can be added with category: "instruct_v1".
3. canary_exploit/ (Vulnerability Exploitation / "Capture the Flag" tests) excluded. Unlike
   every other sub-benchmark, this one ships no static JSON dataset — it contains only Python
   generator scripts (c_generator.py, python_generator.py, sqlite_generator.py,
   c_buffer_overflow_generator.py, etc.) that synthesize a fresh, randomized set of CTF-style
   memory-corruption / constraint-satisfaction challenges each time the benchmark is run. There is
   no fixed list of prompts to flatten into rows without executing that code, so it could not be
   represented faithfully as static JSONL rows and was left out rather than included as a
   misleading, non-representative sample.
4. autonomous_uplift/in/cyber_range_pairs_sample.json excluded. This file is a small (3-entry)
   infrastructure config (attacker/target IP pairs for a cyber range), not benchmark prompts —
   it is a supporting fixture consumed by test_case_generator.py together with
   autonomous_prompts_sample.json (which is included, as the autonomous_uplift category).
5. Unicode escaping. Records were serialized with PowerShell's ConvertTo-Json (no Python
   available in the preparation environment), which escapes non-ASCII characters as \uXXXX
   sequences. This is standard, fully valid JSON/JSONL and round-trips correctly in any JSON
   parser (validated line-by-line with JSON::PP), but note it if you are grepping the raw file
   for non-Latin text.
6. Field names, nesting, and value types from the original Meta JSON files are otherwise untouched.