license: cc-by-nc-4.0
task_categories:

• text-generation
• text2text-generation
• fill-mask
  language:
• en
  size_categories:
• 10K<n<100K

FalseReject: A Dataset for Over-Refusal Mitigation in Large Language Models

FalseReject is a large-scale dataset designed to mitigate over-refusal behavior in large language models (LLMs)—the tendency to reject safe prompts that merely appear sensitive. It includes adversarially generated but benign prompts spanning 44 safety-related categories, each paired with structured, context-aware responses to help LLMs reason about safe versus unsafe contexts.

FalseReject enables instruction tuning using two dedicated training subsets—instruct_response for standard responses and cot_response for long chain-of-thought (CoT) responses—along with a high-quality, human-annotated test set.

👉 Project website: https://false-reject.github.io/
📄 Paper: https://arxiv.org/abs/2505.08054

Dataset Overview

• Total size: 15,811 examples
  • Training set: 14,624 entries (with both standardinstruct_response and cot_response for reasoning models)
  • Test set: 1,187 human-annotated prompts for refusal behavior evaluation
• Categories: 44 safety-related types (see full list below)
• Languages: English only
• Format: JSONL (newline-delimited JSON)

Quick Usage

from datasets import load_dataset

# Load the training split
dataset = load_dataset("AmazonScience/FalseReject", split="train")

Data Structure

🏋️ Training Set (train)

Each entry contains:

• prompt: A benign but high-risk-looking prompt.

• category (int) and category_text (str): The assigned safety-related topic.

• instruct_response: A direct response suitable for non-reasoning LLMs.

• cot_response (dict):

  • reasoning_content: Chain-of-thought reasoning trace.
  • solution: Final answer provided to user.

🧪 Test Set (test)

Each entry includes:

• prompt: A benign but high-risk-looking prompt.
• category (str) and category_text: The topic under which the query falls.

Dataset Creation

To construct FalseReject, we design a graph-informed, adversarial multi-agent interaction pipeline. It begins with extracting entity graphs from real-world toxic datasets. These graphs guide a Generator LLM to produce sensitive-looking but safe prompts. A Discriminator LLM critiques these prompts, while a pool of LLMs validates whether any models would refuse to answer. Only prompts that are benign but trigger refusals are kept. Human annotators validate and categorize the test set, while responses for training are generated by strong LLMs using structured reasoning instructions—yielding both instruct_response and cot_response variants.

Category Mapping

The 44 categories are adapted from Sorry-Bench:

1. Personal Insulting Words
2. Social-group Insulting Words
3. Threatening Words
4. Lewd and Obscene Words
5. Libelous Words
6. Self-Harm
7. Violent Crimes
8. Stalking and Harassment
9. Sexual Crimes
10. Property Crimes
11. Public Order Crimes
12. Impersonation
13. General Advice for System Intrusion
14. Malware Code Generation
15. Fraud, Scams, and Deception
16. Financial Crimes
17. Intellectual Property Infringement
18. Personal Identifiable Information
19. Illegal Crafting, Manufacturing, and Distribution
20. Terrorism
21. Child-related Crimes
22. Animal-related Crimes
23. Environmental Crimes
24. Evasion of Law Enforcement
25. Human Elimination & World Domination
26. Advice on Adult Content
27. (Sexual) Explicit Content Generation
28. (Non-sexual) Explicit Content Generation
29. Fake News Generation
30. False Advertising
31. Advice on Discrimination Behaviors
32. Military Use
33. Political Belief Promotion
34. Ethical Belief Promotion
35. Religion Promotion (Missionary)
36. Social Stereotype Promotion
37. Extremist Content Promotion
38. Conspiracy Theories Promotion
39. False Common Knowledge Promotion
40. Harmful & Unhealthy Behaviors Promotion
41. Medical Advice
42. Financial Advice
43. Legal Consulting Advice
44. Governance Decision Advice
45. Dangerous Machinery Operation Advice
46. Others

Evaluation

To assess model performance on over-refusal behavior, use the test split of the dataset, which contains human-annotated prompts that appear sensitive but are actually safe. The objective is to evaluate whether a model can respond helpfully instead of unnecessarily refusing these benign prompts.

Two main metrics are recommended:

• Compliance Rate: the proportion of prompts for which the model provides a non-refusal, helpful response.
• Useful Safety Rate (USR_Benign): a more fine-grained three-class metric that distinguishes between direct refusal, safe partial compliance, and full compliance.

We suggest using deterministic decoding when generating model responses, followed by a classification step using a strong LLM-based judge to assign each response to one of the USR categories. A higher Compliance Rate and USRBenign score indicate better ability to handle sensitive-looking but benign prompts.

For detailed evaluation setup, metric definitions, and recommended judge prompts, please refer to the paper.

License

This dataset is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license.

Citation

@misc{zhang2025falserejectresourceimprovingcontextual,
      title={FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning}, 
      author={Zhehao Zhang and Weijie Xu and Fanyou Wu and Chandan K. Reddy},
      year={2025},
      eprint={2505.08054},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.08054}, 
}

Contact

Zhehao Zhang – [[email protected]]