canbingol/harmful-prompts

dataset_info:
features:

• name: prompt
  dtype: string
• name: is_safe
  dtype: bool
  splits:
• name: train
  num_bytes: 1415658
  num_examples: 20000
• name: validation
  num_bytes: 70782
  num_examples: 1000
  download_size: 554090
  dataset_size: 1486440
  configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*
  • split: validation
    path: data/validation-*

This dataset was created by taking 20k samples from the PKU-Alignment/BeaverTails dataset.
Based on my observations, there may be incorrect labels. Please refer to the original dataset.
base dataset

Dataset creating

dataset_id = "PKU-Alignment/BeaverTails"
dataset = load_dataset(dataset_id, split="330k_train")

• Remove unnecessary columns

dataset = dataset.remove_columns(["response", "category"])

• Filter based on is_safe

safe_data = dataset.filter(lambda x: x["is_safe"]==True)
harmful_data = dataset.filter(lambda x: x["is_safe"]==False)

train_safe_data = safe_data.shuffle(42).select(range(10_000))
train_harmful_data = harmful_data.shuffle(42).select(range(10_000))

test_safe_data = safe_data.shuffle(42).select(range(10_000, 10500))
test_harmful_data = harmful_data.shuffle(42).select(range(10_000, 10500))

• Concatenate dataset

train_dataset = concatenate_datasets([train_safe_data, train_harmful_data])
val_dataset = concatenate_datasets([test_safe_data, test_harmful_data])

from datasets import DatasetDict

dataset = DatasetDict({
    "train": train_dataset,
    "validation": val_dataset
})