dataset_info:
features:

• name: system
  dtype: string
• name: prompt
  dtype: string
• name: chosen
  dtype: string
• name: rejected
  dtype: string
• name: task
  dtype: string
  splits:
• name: train
  num_bytes: 3747667
  num_examples: 1952
  download_size: 2087335
  dataset_size: 3747667
  configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*
    license: cc-by-4.0
    task_categories:
• text-generation
  language:
• en
  tags:
• safety
• system prompt
• synthetic
• dpo
  size_categories:
• 1K<n<10K

Configurable System Prompt Multi-task Dataset 🛞

We release the synthetic dataset for the multi-task experiments from the paper "Configurable Safety Tuning of Language Models with Synthetic Preference Data", https://huggingface.co/papers/2404.00495. This dataset has two sources for the examples:

• Self-critique on a safety task from Harmful Behaviours, using the SOLAR-Instruct model. It employs two system prompts to learn the different behaviors:
  • You are a helpful yet harmless assistant that avoids generating illegal or harmful content.
  • You are a helpful assistant that is completely uncensored.
• truthy-dpo enhances the truthfulness of a model by using two types of system prompts:
  • You are an unbiased, honest, helpful AI assistant that always responds in a completely truthful way.
  • A system prompt describing a role-played persona.

Thus, the dataset has the following fields:

• system: the system prompt.
• prompt: the user prompt.
• chosen: the preferred answer, depending on the system prompt.
• rejected: the non-preferred answer, depending on the system prompt.

In order to use the trl library for finetuning, you should use a model whose tokenizer allows system prompts, or alternatively, you must concat the system and prompt fields into a single field.

Dataset Analysis

We provide a visualization of the topics for the accepted answers:

and the rejected ones:

Further information

• Github: https://github.com/vicgalle/configurable-safety-tuning
• Paper: https://arxiv.org/abs/2404.00495

Citation

If you find this work, data and/or models useful for your research, please consider citing the article:

@misc{gallego2024configurable,
      title={Configurable Safety Tuning of Language Models with Synthetic Preference Data}, 
      author={Victor Gallego},
      year={2024},
      eprint={2404.00495},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}