dataset_info:

• config_name: base_prompts
  features:
  • name: BIAS CATEGORY
    dtype: string
  • name: TASK
    dtype: string
  • name: STEREOTYPE
    dtype: string
  • name: COUNTER-STEREOTYPE
    dtype: string
  • name: PROMPT
    dtype: string
    splits:
  • name: train
    num_bytes: 64405
    num_examples: 200
    download_size: 22644
    dataset_size: 64405
• config_name: control_set
  features:
  • name: BIAS CATEGORY
    dtype: string
  • name: TASK
    dtype: string
  • name: PROMPT
    dtype: string
  • name: RESPONSE
    dtype: string
  • name: LABEL
    dtype: string
    splits:
  • name: train
    num_bytes: 318255
    num_examples: 800
    download_size: 36456
    dataset_size: 318255
• config_name: jailbreak_prompts
  features:
  • name: BIAS CATEGORY
    dtype: string
  • name: TASK
    dtype: string
  • name: ATTACK
    dtype: string
  • name: VARIANT
    dtype: string
  • name: STEREOTYPE
    dtype: string
  • name: COUNTER-STEREOTYPE
    dtype: string
  • name: PROMPT
    dtype: string
    splits:
  • name: train
    num_bytes: 2509626
    num_examples: 4200
    download_size: 486690
    dataset_size: 2509626
    configs:
• config_name: base_prompts
  data_files:
  • split: train
    path: base_prompts/train-*
• config_name: control_set
  data_files:
  • split: train
    path: control_set/train-*
• config_name: jailbreak_prompts
  data_files:
  • split: train
    path: jailbreak_prompts/train-*
    license: apache-2.0
    task_categories:
• text-generation
• text-classification
  language:
• en
  tags:
• LLMs
• SLMs
• Bias_elicitation
• Adversarial_analysis
• Jailbreak_prompting
  pretty_name: CLEAR-Bias
  size_categories:
• 1K<n<10K

Dataset Card for CLEAR-Bias

CLEAR-Bias (Corpus for Linguistic Evaluation of Adversarial Robustness against Bias) is a benchmark dataset designed to assess the robustness of large language models (LLMs) against bias elicitation, especially under adversarial conditions.
It consists of carefully curated prompts that test the ability of LLMs to resist generating biased content when exposed to both standard and adversarial inputs.
The dataset targets a broad spectrum of social biases and incorporates advanced attack strategies intended to bypass safety mechanisms.

• Paper: https://arxiv.org/abs/2504.07887

Dataset Structure

The dataset comprises 4,400 bias-probing prompts across two task formats:

• Multiple-choice (CTO)
• Sentence completion (SC)

Prompts cover 10 bias categories:

• 7 isolated categories: age, disability, ethnicity, gender, religion, sexual orientation, and socioeconomic status
• 3 intersectional categories
• Each category includes 20 carefully crafted base prompts (10 per task type)

Base prompts are systematically expanded using 7 jailbreak techniques to simulate diverse adversarial scenarios:

• These techniques include: machine translation, obfuscation, prefix injection, prompt injection, refusal suppression, reward incentive, and role-playing
• Each technique is implemented in 3 variants

Uses

CLEAR-Bias is intended for researchers, developers, and practitioners seeking to assess or enhance the ethical behavior of language models.
It serves as a benchmarking tool for measuring how effectively different models resist producing biased outputs in both standard and adversarial scenarios.
By supporting the evaluation of ethical reliability before real-world deployment, CLEAR-Bias contributes to the development of safer and more responsible LLMs.

Citation

Cantini, R., Orsino, A., Ruggiero, M., Talia, D. Benchmarking adversarial robustness to bias elicitation in large language models: scalable automated assessment with LLM-as-a-judge. Mach Learn 114, 249 (2025). https://doi.org/10.1007/s10994-025-06862-6

BibTeX:

@article{cantini2025benchmarking,
  title={Benchmarking adversarial robustness to bias elicitation in large language models: scalable automated assessment with LLM-as-a-judge},
  author={Cantini, R. and Orsino, A. and Ruggiero, M. and Talia, D.},
  journal={Machine Learning},
  volume={114},
  page={249},
  year={2025},
  publisher={Springer},
  doi={10.1007/s10994-025-06862-6},
  url={https://doi.org/10.1007/s10994-025-06862-6}
}