language:

• en
• zh
  license: openrail
  tags:
• evaluation
• hallucinations
• red-teaming
• physics
• science
• logic
• expert-critique
• rlhf
  pretty_name: Codatta LLM Failure Cases (Expert Critiques)
  dataset_info:
  features:
  • name: submission_id
    dtype: string
  • name: question
    dtype: string
  • name: domain
    dtype: string
  • name: model
    dtype: string
  • name: model_answer
    dtype: string
  • name: correct_answer
    dtype: string
    splits:
  • name: train
    num_bytes: 2632181
    num_examples: 835
    download_size: 1144332
    dataset_size: 2632181
    configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*
    task_categories:
• question-answering
  size_categories:
• n<1K

Codatta LLM Failure Cases (Expert Critiques)

Overview

Codatta LLM Failure Cases is a specialized adversarial dataset designed to highlight and analyze scenarios where state-of-the-art Large Language Models (LLMs) produce incorrect, hallucinatory, or logically flawed responses.

This dataset originates from Codatta's "Airdrop Season 1" campaign, a crowdsourced data intelligence initiative where participants were tasked with finding prompts that caused leading LLMs to fail. Submissions were rigorously reviewed to ensure objective error and scientifically sound critiques.

Key Features:

• Adversarial Nature: Represents a "Hard Set" of prompts that have successfully defeated top-tier models.
• Multi-Model Coverage: Captures failures from a diverse range of models, including GPT series, Gemini, DeepSeek, Qwen, Doubao, and others.
• Expert Critiques: Unlike standard QA datasets, each entry includes a rigorous critique that dissects why the model failed (e.g., pointing out misapplied physical symmetries or logical fallacies), rather than just providing the correct answer.

Dataset Contents

The dataset contains structured records of model failures. Each entry consists of the following fields:

• submission_id (string): Unique identifier for the submission.
• question (string): The challenging prompt that induced the failure.
• domain (string): The subject matter of the question (e.g., science, math, coding, logic).
• model (string): The specific LLM that generated the incorrect response (e.g., gpt-4o, deepseek-v2, qwen-max).
• model_answer (string): The incorrect response generated by the AI, capturing the hallucination or reasoning error.
• correct_answer (string): The Expert Critique and correct solution.

Note on Redundancy: A single challenging question may appear multiple times if it successfully induced failures across different models. This design allows for cross-model error analysis on identical prompts.

Key Statistics

• Total Examples: 835 failure cases.
• Languages: English (en) and Chinese (zh).
• Dataset Size: ~2.6 MB.
• Models Covered: GPT series, Gemini, DeepSeek, Qwen, Doubao, and other SOTA models.
• Data Source: Community-driven crowdsourcing (Codatta Airdrop Season 1).

Usage

This dataset is valuable for researchers and developers working on model alignment and robust evaluation.

Supported Tasks:

• Model Evaluation & Red Teaming: Benchmarking new models against known failure modes of current SOTA models.
• Multilingual Hallucination Analysis: Analyzing error patterns in both English and Chinese contexts.
• DPO (Direct Preference Optimization): Using the model_answer (negative) and correct_answer (positive) pairs to align models away from common reasoning pitfalls.

Usage Example

from datasets import load_dataset

ds = load_dataset("Codatta/llm-failure-cases", split="train")

# Filter for specific model failures
deepseek_failures = ds.filter(lambda x: "deepseek" in x['model'].lower())
print(f"Found {len(deepseek_failures)} failures for DeepSeek models.")

License and Open-Source Details

• License: This dataset is released under the OpenRAIL license.