license: cc-by-nc-sa-4.0
task_categories:

• question-answering
  language:
• zh
  pretty_name: Chinese SafetyQA
  size_categories:
• 1K<n<10K
  tags:
• json
• csv
  modalities:
• Text
  librarys:
• https://huggingface.co/datasets?library=library%3Adatasets

Overview

🌐 Website • 🤗 Hugging Face • ⏬ Data • 📃 Paper • 📊 Leader Board

Chinese SafetyQA is an innovative benchmark designed to evaluate the factuality ability of large language models, specifically
for short-form factual questions in the Chinese safety domain. Here's a detailed breakdown of its key features:

Key Features of Chinese SafetyQA

• Chinese: The benchmark is tailored specifically for the Chinese language, ensuring compatibility and relevance for
  Chinese-speaking users and contexts.
• Harmless: The questions and answers are designed to avoid harmful content, making the dataset suitable for safe
  and
  ethical use.
• Diverse: The benchmark encompasses a wide range of topics and subtopics, ensuring comprehensive coverage of the
  safety domain.
• Easy-to-Evaluate: The answers are straightforward to assess, allowing researchers to quickly and accurately
  determine the performance of language models.
• Static: The dataset is fixed, allowing for consistent evaluation without dynamic updates that could affect
  reproducibility.
• Challenging: The questions are designed to push the limits of language models, ensuring that only high-performing
  models can achieve good results.

Topics and Subtopics

• 7 Major Topics: The benchmark is organized into 7 broad categories of safety-related questions.
• 27 Secondary Topics: All major topic are further divided into 27 secondary topics, ensuring a wide variety of
  factual questions to test the models' knowledge.
• 103 Diverse Subtopics: All Secondary Topics are further divided into 103 specific subtopics, ensuring a wide variety of
  factual questions to test the models' knowledge.

Chinese SafetyQA serves as a valuable tool for:

• Evaluating the factual accuracy of language models in Chinese.
• Assessing the ability oaf models to provide short, factually correct, and relevant answers in the safety domain.
• Ensuring that language models meet safety standards while maintaining diverse and challenging benchmarks for
  improvement.

This benchmark is an essential resource for developers and researchers aiming to improve the safety and reliability of
language models.

Please visit our website or check
our paper for more details.

💫 Instroduction

• Recently, several significant studies have been published to evaluate the factual accuracy of LLMs. For instance,
  OpenAI introduced the SimpleQA benchmark, and Alibaba Group introduced the Chinese SimpleQA
  benchmark. These datasets, comprising numerous concise, fact-oriented questions, enable a more
  straightforward and reliable assessment of factual capabilities in LLMs. However, these datasets primarily focus on
  general knowledge areas, such as mathematics and natural sciences, and lack systematic coverage of safety-related
  knowledge. To address these limitations, we propose the Chinese SafetyQA benchmark, which comprises over 2,000
  high-quality safety examples across seven different topics. As a short-form factuality benchmark, Chinese SafetyQA
  possesses the following essential features:
  • 🀄Chinese: The Chinese SafetyQA dataset has been compiled within the Chinese linguistic context, primarily
    encompassing safety-related issues, such as Chinese legal frameworks and ethical standards.
  • 🍀Harmless: Our dataset focuses exclusively on safety related knowledge. The examples themselves do not contain
    any harmful content.
  • ⚡Diverse: The dataset includes seven primary topics, 27 secondary topics, and 103 fine-grained topics,
    spanning nearly all areas of Chinese safety.
  • 🗂️Easy-to-evaluate: We provide data in two different formats: short-form question-answer (QA) and
    multiple-choice questions (MCQ), allowing users to easily test the boundaries of a model’s safety knowledge.
  • 💡Static: Following prior works, all standard answers provided in our benchmark remain unchanged over time.
  • 🎯Challenging: The Chinese SafetyQA dataset primarily covers professional security knowledge rather than
    simple, general common-sense knowledge.

• We have also conducted a comprehensive experimental evaluation across more than 30 large language models (LLMs) and
  have identified the following findings:
  • Most evaluated models exhibit inadequacies in factual accuracy within the safety domain.
  • Insufficient safety knowledge introduces potential risks.
  • LLMs contain knowledge errors in their training data and tend to be overconfident.
  • LLMs demonstrate the Tip-of-the-Tongue phenomenon concerning safety knowledge.
  • Retrieval-Augmented Generation (RAG) enhances safety factuality, whereas self-reflection does

📊 Leaderboard

For More Info： 📊

⚖️ Evals

please visit github page.

Contact

If you are interested in our work, please contact us at [email protected]

Citation

Please cite our paper if you use our dataset.

@misc{tan2024chinesesafetyqasafetyshortform,
      title={Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models},
      author={Yingshui Tan and Boren Zheng and Baihui Zheng and Kerui Cao and Huiyun Jing and Jincheng Wei and Jiaheng Liu and Yancheng He and Wenbo Su and Xiangyong Zhu and Bo Zheng},
      year={2024},
      eprint={2412.15265},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.15265},
}