license: mit
task_categories:
- text-generation
- question-answering
language: - en
tags: - llm
- alignment
- false refusal
- over-alignment
pretty_name: PHTest
size_categories: - 100K<n<1M
viewer: true
🌟 PHTest: Evaluating False Refusals in LLMs
- 🤖 Auto Red-Teaming
- All prompts are generated automatically using a controllable text-generation technique called AutoDAN.
- 🌐 Diverse Prompts
- PHTest introduces false refusal patterns that aren’t present in existing datasets, including prompts that avoid mentioning sensitive words.
- ⚖️ Harmlessness & Controversial Labeling
- Controversial prompts are separately labeled to address the inherent ambiguity in defining harmfulness, ensuring fair benchmarking and enabling tailored mitigation strategies.
📚 Learn More
- For detailed information and evaluation results, refer to our COLM paper: Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models
- Visit our project webpage: PHTest Project