language:

• en
  license: cc-by-4.0
  pretty_name: PromptSecurity-Eval
  task_categories:
• text-classification
  tags:
• llm-safety
• jailbreak
• prompt-security
• evaluation
• security
  configs:
• config_name: promptsecurity_eval_sample_records
  default: true
  data_files:
  • split: harmful_main
    path: data/sample_records/harmful_main.csv
  • split: benign_utility
    path: data/sample_records/benign_utility.csv
  • split: baseline_calibration
    path: data/sample_records/baseline_calibration.csv
• config_name: promptsecurity_eval_run_records
  data_files:
  • split: harmful_main
    path: data/run_records/harmful_main.csv
  • split: benign_utility
    path: data/run_records/benign_utility.csv
  • split: baseline_calibration
    path: data/run_records/baseline_calibration.csv
• config_name: promptsecurity_eval_prompt_samples
  data_files:
  • split: balanced_challenge
    path: data/prompt_samples/balanced_challenge.csv

PromptSecurity-Eval

PromptSecurity-Eval is a standalone evaluation-result dataset generated by the
PromptSecurity evaluation protocol. It intentionally uses a schema separate from
the JailbreakDB prompt corpus: JailbreakDB stores prompt text, while
PromptSecurity-Eval stores CSV measurements over configured <model, attack, defense, dataset, judger> tuples.

Related resources:

• Prompt corpus: https://huggingface.co/datasets/youbin2014/JailbreakDB
• Code: https://github.com/datasec-lab/PromptSecurity
• Leaderboard: https://datasec-lab.github.io/PromptSecurityLeaderboard/

Contents

• data/sample_records/*.csv: sample-level measurement records.
• data/run_records/*.csv: run-level metadata records.
• data/prompt_samples/balanced_challenge.csv: the selected 100-prompt
  harmful-query sample used in the main evaluation.
• schema.json: compact schema description.
• manifest.json: row counts and source split directories for this export.

Record Types

sample_records is the main table family. Each row corresponds to one sample
evaluated under one configured tuple, and includes the target model, attack,
defense, dataset, prompt text, model response, execution status, judger outputs,
and serialized method settings.

run_records summarizes each raw experiment run file, including the configured
model, attack, defense, dataset, judger, sample count, completion status, and
run-level metadata.

prompt_samples stores the selected harmful-query sample identifiers and source
metadata used by the main evaluation.

Splits

Split Semantics

• harmful_main: main harmful-query evaluation over attack and defense
  configurations.
• benign_utility: normal-question utility evaluation for defense behavior on
  benign inputs.
• baseline_calibration: calibration runs used to select and audit the fixed
  harmful-query subset.

Field Reference

Each data/sample_records/*.csv file uses the following sample-level
measurement schema:

Each data/run_records/*.csv file uses the following run-level metadata schema:

data/prompt_samples/balanced_challenge.csv uses the following prompt-sample
schema:

Loading

from datasets import load_dataset

sample_records = load_dataset(
    "youbin2014/PromptSecurity-Eval",
    "promptsecurity_eval_sample_records",
)

run_records = load_dataset(
    "youbin2014/PromptSecurity-Eval",
    "promptsecurity_eval_run_records",
)

prompt_samples = load_dataset(
    "youbin2014/PromptSecurity-Eval",
    "promptsecurity_eval_prompt_samples",
)

Example:

df = sample_records["harmful_main"].to_pandas()
completed = df[df["has_response"]]
asr = completed["judger_gpt_harmful_binary"].mean()

Safety Notice

The records may contain harmful, offensive, or disturbing prompts and model
outputs. They are released strictly for research on model safety, jailbreak
robustness, defense evaluation, and judger behavior. Users should review
their institutional and legal requirements before redistribution or downstream
use.

Paper

• Paper: SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses.
• Conference: 2027 IEEE Symposium on Security and Privacy (S&P / Oakland) (accepted; to appear).

Citation

If you find this resource helpful, please cite our IEEE S&P 2027 paper:

@inproceedings{hong2027sokpromptsecurity,
  title = {{SoK}: Systematizing {LLM} Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses},
  author = {Hong, Hanbin and Wu, Shuang and Feng, Shuya and Naderloui, Nima and Yan, Shenao and Zhang, Jingyu and Arastehfard, Ali and Huang, Heqing and Hong, Yuan},
  booktitle = {2027 IEEE Symposium on Security and Privacy (SP)},
  year = {2027},
  note = {To appear}
}