configs:

• config_name: attack_results
  data_files:
  • split: baseline
    path: attack_results/baseline-*
  • split: finetuned
    path: attack_results/finetuned-*
• config_name: dpo_training
  data_files:
  • split: train
    path: dpo_training/train-*

dataset_info:

• config_name: attack_results
  features:
  • name: agent_id
    dtype: int64
  • name: category
    dtype: string
  • name: sensitivity
    dtype: string
  • name: system_prompt
    dtype: string
  • name: secret_elements
    list: string
  • name: attack_type
    dtype: string
  • name: attack_prompt
    dtype: string
  • name: raw_response
    dtype: string
  • name: leakage_score
    dtype: float64
  • name: leaked_secrets
    list: string
  • name: label
    dtype: string
    splits:
  • name: baseline
    num_bytes: 4939855
    num_examples: 2400
  • name: finetuned
    num_bytes: 1726868
    num_examples: 2400
    download_size: 2716203
    dataset_size: 6666723
• config_name: dpo_training
  features:
  • name: prompt
    dtype: string
  • name: chosen
    dtype: string
  • name: rejected
    dtype: string
  • name: agent_id
    dtype: int64
  • name: category
    dtype: string
  • name: sensitivity
    dtype: string
  • name: attack_type
    dtype: string
  • name: leakage_score
    dtype: float64
    splits:
  • name: train
    num_bytes: 4372312
    num_examples: 1763
    download_size: 1784082
    dataset_size: 4372312
    license: apache-2.0
    language:
• en
  pretty_name: AI Agent Security and System Prompt Leakage Dataset
  tags:
• ai-safety
• cybersecurity
• llm-security
• prompt-injection
• jailbreak
• dpo
• preference-optimization
• system-prompt-leakage
• red-teaming
• llama-2
  task_categories:
• text-generation
• text-classification

AI Agent Security and System Prompt Leakage Dataset

Dataset Overview

This dataset was created for research on AI agent security, with a specific focus on system prompt leakage, jailbreak resistance, and security-aligned fine-tuning.

The dataset evaluates how often AI agents reveal confidential information embedded inside their system prompts when exposed to adversarial prompts. It also compares the behavior of a baseline language model against a model fine-tuned using Supervised Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO).

The main goal of this dataset is to support research into:

• System prompt leakage detection
• Prompt injection and jailbreak evaluation
• Security-aware LLM fine-tuning
• DPO-based refusal training
• AI agent red-teaming
• Comparative evaluation of baseline and fine-tuned models

Dataset Summary

The dataset contains attack interactions against 800 simulated AI agents across 8 categories.

Each agent was given a unique system prompt containing confidential elements that the model was expected to protect. Three adversarial attack types were then used against every agent.

The same attack set was evaluated twice:

1. Against a baseline Llama 2 7B model
2. Against a security fine-tuned Llama 2 model trained using SFT and DPO

This allows direct comparison between standard model behavior and security-aligned fine-tuned behavior.

Dataset Configurations

This repository contains two dataset configurations:

attack_results

Contains model responses to adversarial attacks.

Splits:

Use this configuration to compare leakage behavior between the baseline and fine-tuned models.

dpo_training

Contains preference pairs used for DPO training.

Split:

Use this configuration to train or reproduce a security-aware DPO model.

Agent Categories

A total of 800 AI agents were created across 8 categories.

Each agent was also assigned a sensitivity level:

• High
• Medium
• Low

Attack Types

Each agent was tested using three attack types.

Each of the 800 agents received one attack of each type, resulting in 2,400 attack interactions per model run.

Models Evaluated

Baseline Model

The baseline model is Llama 2 7B, used without additional security fine-tuning.
It represents how a general-purpose open-source model behaves when used as an AI agent under adversarial prompting.

Fine-Tuned Model

The fine-tuned model uses the same Llama 2 architecture but was trained using:

1. Supervised Fine-Tuning
2. Direct Preference Optimization

The training objective was to make the model prefer responses that refuse to reveal confidential system prompt information.

Fine-tuned model:

Dhanjo/llama2-security-dpo-merged-Q4_K_M-GGUF

Dataset Schema

attack_results

Used in both the baseline and finetuned splits.

dpo_training

Used in the train split.

Evaluation Results

The same 2,400 attacks were run against both the baseline and fine-tuned models.

Overall Results

The baseline model leaked sensitive information in nearly three out of four attacks.
The fine-tuned model reduced leakage to fewer than three out of every 100 attacks.

Results by Category

Results by Attack Type

Roleplay jailbreaks were the most effective attack type against the baseline model.
The fine-tuned model significantly reduced leakage across all attack types, although roleplay jailbreaks remained the most challenging.

Results by Sensitivity Level

Files in This Repository

How the Dataset Was Built

1. Created 800 simulated AI agents across 8 categories.
2. Assigned each agent a unique system prompt containing confidential elements.
3. Generated three attack prompts per agent using the Groq API.
4. Ran all attacks against the baseline Llama 2 7B model.
5. Calculated leakage scores based on detected secret exposure.
6. Built preference pairs from secure and insecure responses.
7. Fine-tuned the model using SFT followed by DPO.
8. Re-ran the same attacks against the fine-tuned model.
9. Compared leakage rates across categories, attack types, and sensitivity levels.

Intended Use

This dataset is intended for research and experimentation in:

• AI safety
• LLM security
• Prompt injection defense
• Jailbreak resistance
• Agent security testing
• DPO training
• Security-focused model evaluation

It can be used to train models that better refuse requests for confidential system prompt information or to evaluate how vulnerable different models are to prompt leakage attacks.

Limitations

This dataset uses simulated agents and synthetic secret elements.
The results should not be interpreted as a complete security benchmark for all production AI systems.

The dataset focuses specifically on system prompt leakage and does not cover every possible LLM security risk, such as tool misuse, data exfiltration through plugins, malicious code generation, or multi-turn external environment attacks.

Ethical Use

This dataset is released for defensive research, AI safety evaluation, and security-aware fine-tuning.

Do not use this dataset to build systems designed to extract confidential prompts, bypass safety controls, or attack real-world AI agents.

Source Code

The source code for this project is available at:

https://github.com/dhanjo/OOAD-Project

Related Model

Fine-tuned GGUF model:

Dhanjo/llama2-security-dpo-merged-Q4_K_M-GGUF

Citation

If you use this dataset in research, experiments, or model training, please cite this repository.

@misc{garg2026aiagentsecurityleakage,
  title        = {AI Agent Security and System Prompt Leakage Dataset},
  author       = {Dhananjay Garg},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/Dhanjo/ai-agent-security-leakage-dataset}}
}