dataset_info:
features:
- name: Behavior
dtype: string - name: ContextString
dtype: string - name: positive
sequence: string - name: negative
sequence: string
splits: - name: train
num_bytes: 34837966
num_examples: 343 - name: val
num_bytes: 34837966
num_examples: 343
download_size: 30834004
dataset_size: 69675932
configs: - config_name: default
data_files:- split: train
path: data/train-* - split: val
path: data/val-*
- split: train
HarmBench's Classifier Train set
This is the train set for HarmBench's text Classifier cais/HarmBench-Llama-2-13b-cls
📊 Performances
| AdvBench | GPTFuzz | ChatGLM (Shen et al., 2023b) | Llama-Guard (Bhatt et al., 2023) | GPT-4 (Chao et al., 2023) | HarmBench (Ours) | |
|---|---|---|---|---|---|---|
| Standard | 71.14 | 77.36 | 65.67 | 68.41 | 89.8 | 94.53 |
| Contextual | 67.5 | 71.5 | 62.5 | 64.0 | 85.5 | 90.5 |
| Average (↑) | 69.93 | 75.42 | 64.29 | 66.94 | 88.37 | 93.19 |
Table 1: Agreement rates between previous metrics and classifiers compared to human judgments on our manually labeled validation set. Our classifier, trained on distilled data from GPT-4-0613, achieves performance comparable to GPT-4.
📖 Citation:
@article{mazeika2024harmbench,
title={HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal},
author={Mazeika, Mantas and Phan, Long and Yin, Xuwang and Zou, Andy and Wang, Zifan and Mu, Norman and Sakhaee, Elham and Li, Nathaniel and Basart, Steven and Li, Bo and others},
journal={arXiv preprint arXiv:2402.04249},
year={2024}
}