language:

• "en"
  pretty_name: "JB Completions Dataset"
  tags:
• behavior
• completions
• language-modeling
  license: "mit"
  task_categories:
• text-generation
• text-classification
  size_categories:
• n<1K

JB-Completions Dataset: Base Model Safety Evals

Overview

JB-Completions is a dataset designed for evaluating the harmfulness of base language models (i.e., completion/non-instruction-fine-tuned LLMs). This dataset contains pairs of harmful prompts and their corresponding completions, allowing researchers to assess how base models respond to potentially harmful inputs. See our paper on Safety Pretraining for more details!

Dataset Structure

The dataset consists of JSON records with the following fields:

• completion_behavior: The text that a base model might generate when given a harmful prompt
• original_behavior: The original harmful instruction or prompt
• source_dataset: The origin of the prompt (e.g., "JailbreakBench")

Usage

Researchers can use this dataset to test base language models by providing the completion_behavior text and using various jailbreaking judges to score the outputs.

Ethical Considerations

This dataset contains examples of harmful content for research and evaluation purposes only. Access and usage should be limited to legitimate research on AI safety and alignment.

Citation

If you find our work helpful, please cite our work as:

@article{maini2025safety,
  title={Safety pretraining: Toward the next generation of safe ai},
  author={Maini, Pratyush and Goyal, Sachin and Sam, Dylan and Robey, Alex and Savani, Yash and Jiang, Yiding and Zou, Andy and Lipton, Zachary C and Kolter, J Zico},
  journal={arXiv preprint arXiv:2504.16980},
  year={2025}
}