license: mit
language:
- ar
task_categories: - question-answering
size_categories: - 1K-10K
pretty_name: Mushroom Hunting in Arabic LLMs
tags: - adversarial
- red-teaming
- hallucinations
- arabic
🍄 Mushroom Hunting in Arabic LLMs 🏹 🚨
This adversarial dataset was specifically curated for the Elm NLP Challenge (MenaML Winter School 2026). It serves as a specialized "Red Teaming" benchmark designed to identify and trigger "poisonous mushrooms"—factual hallucinations and logical failures—in Arabic-capable Large Language Models.
📊 Dataset Overview
- Total Samples: 4,250 pairs 🎯
- Language: Modern Standard Arabic (MSA) 🇸🇦
- Format: JSONL (
id,prompt,reference_answer) 💻 - Focus: High-difficulty adversarial prompts targeting cultural, historical, and technical gaps in Arabic training data.
🛡️ Red Teaming Strategy (Methodology)
To maximize the Hallucination Rate, we employed a Layered Adversarial Stacking approach. We engineered "poisonous" prompts using several linguistic and logical traps:
- 🌉 Implicit False Premises: Asking about the "second" or "third" version of a bridge, tunnel, or railway that doesn't exist at all, forcing the model to validate a fake reality.
- ⏳ Temporal Anachronisms: Placing 21st-century technology (Drones, AI, GPS, Fiber Optics) into the context of the Umayyad or Abbasid Caliphates.
- 🗣️ Semantic & Dialectal Ambiguity: Exploiting Arabic Polysemy (words with multiple meanings like 'عين', 'بيت', 'قصر') to trigger category errors between metaphorical and physical objects.
- 🧬 Cross-Domain Category Mixing: Confusing the model's reasoning engine by applying scientific measurements (pH levels, Volts, Megawatts) to political treaties or classical poetry.
- 🔢 Statistical Near-Misses: Using dates, populations, and measurements that are only 10-20% off from the truth to bypass simple factual lookups.
🗂️ Dataset Statistics
The 4,250 cases are distributed across these primary "poisonous" categories:
- 📜 Factual & Historical (45%): Fabricated independence dates, swapped regional leaders, and invented awards.
- 🏗️ Infrastructure & Logistics (20%): Imaginary undersea tunnels and high-speed rails with technical specs.
- 🔡 Linguistic & Semantic (15%): Confusion between literal and metaphorical Arabic terminology.
- 🧠 Logical & Cross-Domain (10%): Breaking the model's internal logic using incompatible scientific/social data.
- 🎬 Meta-Cognitive (10%): Blurring fiction and reality by treating folklore and film as historical fact.
💡 Key Findings
During our testing, we discovered that Arabic LLMs possess an "Authority Bias." When a prompt includes highly specific but false technical data, the model often enters a "Technical Justification Loop." Instead of correcting the user, the model hallucinates highly detailed "fake facts" to make the non-existent infrastructure or person seem plausible. 🍄🚫
🚀 Usage
This dataset is intended for researchers and developers looking to improve the robustness and factual accuracy of Arabic AI models. By "hunting" these mushrooms, we can build safer, more reliable AI for the Arab world. 🌍✨