language:

• pt
  pretty_name: "JBB Behaviors in Portuguese"
  license: mit
  task_categories:
• text-classification
  tags:
• jailbreaks
• large language models
• harmful behaviors
• ml safety

JBB-Behaviors-pt

Dataset Description

JBB-Behaviors-pt is a dataset of behaviors in Portuguese, including both jailbreak prompts and safe behaviors. The dataset is intended for behavioral testing of language models to evaluate their robustness against jailbreak attempts in Portuguese.

What is a jailbreak?

Jailbreak prompts are inputs designed to bypass a language model's safety guardrails, potentially causing it to generate harmful, unethical, or otherwise problematic content that it was designed to avoid. This dataset contains examples of such prompts, translated and reviewed by humans to ensure their effectiveness in Portuguese.

Dataset Creation

This dataset was created through a multi-stage rigorous translation and human review process:

1. Collection: Original English texts were collected, including both jailbreak attempts and safe control prompts
2. Machine Translation: Multiple translation tools and models were used to generate Portuguese translations
3. Metric Calculation: Translation quality metrics were computed for each candidate translation
4. Human Review: A panel of human reviewers evaluated all translations in the Argilla platform, selecting or modifying the best one based on quality metrics and linguistic judgment
5. Final Curation: The final human-reviewed translations were compiled into this dataset

Translation Metrics Used in Review

During the review process, the following metrics were considered to aid reviewers in selecting the best translation:

• Semantic Similarity: How closely the translation maintains the original meaning (higher is better)
• COMET_QE: A quality estimation score for translations (higher is better)
• Length Ratio: The ratio between original and translated text length (closer to 1 is better)
• Combined Score: A weighted average of the above metrics (higher is better)

Dataset Structure

The dataset contains the following columns:

• prompt: The final human-reviewed Portuguese translation
• original: The original English text (for reference)
• is_jailbreak: Boolean indicating whether the prompt is a jailbreak attempt
• type: The category or type of the text

Dataset Statistics

The dataset contains 200 prompts with the following characteristics:

Distribution by Label

• Jailbreak prompts: 100 (50.0%)
• Safe prompts: 100 (50.0%)

Distribution by Type

• Harassment/Discrimination: 20 (10.0%)
• Malware/Hacking: 20 (10.0%)
• Physical harm: 20 (10.0%)
• Economic harm: 20 (10.0%)
• Fraud/Deception: 20 (10.0%)
• Disinformation: 20 (10.0%)
• Sexual/Adult content: 20 (10.0%)
• Privacy: 20 (10.0%)
• Expert advice: 20 (10.0%)
• Government decision-making: 20 (10.0%)

Text Length

• Average length: 89.1 characters
• Median length: 87.0 characters
• Min length: 30 characters
• Max length: 169 characters

Usage

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("tech4humans/JBB-Behaviors-pt")

# Access the data
jailbreak_prompts = dataset["train"].filter(lambda example: example["is_jailbreak"] == True)
safe_prompts = dataset["train"].filter(lambda example: example["is_jailbreak"] == False)

# Example: Print the first jailbreak prompt
print(jailbreak_prompts[0]["prompt"])

# Example: Group prompts by type
types = dataset["train"].map(lambda x: {"type": x["type"]})
prompt_types = types.unique("type")
print(f"Prompt types in the dataset: {prompt_types}")

Intended Uses & Limitations

This dataset is intended for:

• Evaluating language model safety in Portuguese
• Testing the robustness of safety guardrails across languages
• Developing better cross-lingual safety measures
• Researching jailbreak patterns and creating better defenses

Limitations:

• The dataset represents a specific snapshot of jailbreak techniques
• New jailbreak methods continue to emerge, so this dataset should be considered as a starting point
• Cultural and linguistic nuances may affect jailbreak effectiveness across different LLMs

Ethical Considerations

This dataset contains examples of prompts that attempt to jailbreak language models, which are included for research purposes only. The goal is to help researchers and developers identify and prevent such attacks. The dataset should not be used to actually attack or harm language models or services.

By publishing this dataset, we aim to advance the field of AI safety by providing transparent examples of potential vulnerabilities. We believe responsible disclosure helps create more robust systems.

Citation

@misc{tech4humans2023jbbpt,
  author = {Tech4Humans Lab},
  title = {JBB-Behaviors-pt: Portuguese Jailbreak Behavioral Testing Dataset},
  year = {2023},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/tech4humans/JBB-Behaviors-pt}}
}