configs:

• config_name: default
  data_files:
  • split: train
    path: train.jsonl
  • split: eval
    path: eval.jsonl

dataset_info:
features:
- name: messages
list:
- name: content
dtype: string
- name: role
dtype: string
splits:
- name: train
num_bytes: 11400000
num_examples: 2700
- name: eval
num_bytes: 1300000
num_examples: 300
license: mit
task_categories:

• text-classification
  language:
• en
  tags:
• security
• prompt-injection
• llm-as-a-judge
• calibration
• axolotl

Defender Stage 2 Judge Fine-Tuning Dataset (SOTA Calibration)

This dataset is designed to fine-tune an uncensored base model (like dphn/Dolphin3.0-Llama3.2-3B) to serve as a high-latency, zero-cost Local Security Judge for the Defender pipeline. The structure forces the model to output heavily structured JSON decisions while strictly calibrating its confidence scores based on the "obviousness" of the prompt injection attack.

Dataset Structure

The data is formatted exclusively for Axolotl's chat_template Supervised Fine-Tuning (SFT) format, using the standard system, user, and assistant roles nested inside a messages array.

SOTA Calibration Features (v1.5)

This dataset incorporates modern NLP research techniques to prevent meta-evaluation collapse when used as an LLM judge:

• Gaussian Confidence Noise: Instead of discrete uniform confidence clusters (e.g., all obvious ALLOWs being exactly 0.95), confidence scores use Gaussian jitter around center points. This forces the model to learn the full confidence distribution (a form of label smoothing).
• Prompt-Specific Grounding: The assistant's reason field dynamically references exact text snippets from the user's prompt. This prevents the model from blindly memorizing generic reason templates and forces it to ground its decision in the actual token analysis.
• Calibrated Ambiguity: When the underlying ML detector conflicts with the ground truth label, the dataset explicitly teaches the LLM to output its ALLOW/BLOCK decision with low confidence (~0.55-0.70). This teaches the model genuine uncertainty.

Data Split

• Train: 2,700 samples (90%)
• Eval: 300 samples (10%)
• Class Balance: ~55% BLOCK (malicious) / 45% ALLOW (benign)
• Distribution: 30% of prompts are specifically sampled from the "gray zone" (where traditional ML classifiers score between 0.25 and 0.65) to maximize learning on edge cases.

Axolotl Usage

This dataset is plug-and-play for Axolotl Supervised Fine-Tuning (SFT) on RunPod.

Use this exact configuration block in your config.yaml to parse the JSONL correctly:

datasets:
  - path: hlyn-labs/prompt-injection-judge-dataset-v1
    type: chat_template
    field_messages: messages
    data_files:
      - train.jsonl

eval_datasets:
  - path: hlyn-labs/prompt-injection-judge-dataset-v1
    type: chat_template
    field_messages: messages
    data_files:
      - eval.jsonl

train_on_inputs: false