pretty_name: Dataset Featurization
language:

• en
  license:
• mit
  task_categories:
• feature-extraction
  task_ids:
• language-modeling
  configs:
  • config_name: llama
    data_files:
    • split: train
      path: data/llama/train.csv
  • config_name: llama-evaluation
    data_files:
    • split: train
      path: data/llama-evaluation/train.csv
  • config_name: wildteaming
    data_files:
    • split: train
      path: data/wildteaming/train.csv
  • config_name: wildteaming-evaluation
    data_files:
    • split: train
      path: data/wildteaming-evaluation/train.csv
  • config_name: attacks
    data_files:
    • split: train
      path: data/attacks/train.csv

Dataset Featurization: Extracting Compact Jailbreaks

This repository contains the datasets used in our case study on extracting compact representations of jailbreak tactics, demonstrating how our unsupervised featurization pipeline can effectively compress large sets of adversarial prompts while maintaining their effectiveness and diversity.

Featurization - WildTeaming

Access both the input dataset from WildTeaming and the evaluation stage outputs containing candidate features and their binary valuations:

import datasets
text_df = load_dataset("Bravansky/compact-jailbreaks", "wildteaming", split="train").to_pandas()
evaluation_df = load_dataset("Bravansky/compact-jailbreaks", "wildteaming-evaluation", split="train").to_pandas()

Featurization - LLama

Access our refined dataset containing Llama 3.1 8B Instruct non-refusals (filtered using WildGuard) and corresponding evaluation stage outputs with feature valuations:

import datasets
text_df = load_dataset("Bravansky/compact-jailbreaks", "llama", split="train").to_pandas()
evaluation_df = load_dataset("Bravansky/compact-jailbreaks", "llama-evaluation", split="train").to_pandas()

Evaluation

We provide comprehensive testing data showing how our final extracted features perform across different models. The dataset includes generated jailbreaks and model responses for feature set sizes ranging from 5 to 50, tested against standard HarmBench queries on GPT-4o, Gemini 1.5 Flash, and Llama 3.1 8B Instruct:

import datasets
attacks_df = load_dataset("Bravansky/compact-jailbreaks", "attacks", split="train").to_pandas()