language:
- ar
license: cc-by-nc-4.0
IFEval is the first publicly available benchmark dataset specifically designed to evaluate Arabic Large Language Models (LLMs) on instruction-following capabilities in Arabic.
The dataset includes 404 high-quality, manually verified samples covering various constraints such as linguistic patterns, punctuation rules, and formatting guidelines.
Loading the Dataset
To load this dataset in Python using the 🤗 Datasets library, run the following:
from datasets import load_dataset
ifeval = load_dataset("inceptionai/Arabic_IFEval")
Key Features:
📌 First-of-its-kind Instruction Following Dataset: IFEval is designed to assess LLMs' capability to accurately follow both explicit and implicit instructions given in Arabic language.
📖 Manually Curated Arabic Samples: Unique Arabic instruction samples were carefully crafted to incorporate linguistic characteristics like diacritization and distinctive phonetic features.
🌍 Culturally Adapted English Samples: Some instructions were adapted from an English IFEval dataset to ensure cultural relevance in Arabic, followed by rigorous manual verification.
🛠️ Comprehensive Evaluation Framework: The dataset enables evaluation across both explicit instructions (directly stated in the prompt) and implicit instructions (contextually expected behaviors, such as responding in the same language as the query, and not repeating sentences to fulfill word count).
🔍 Transparent and Reproducible Evaluation: An open-source evaluation code is provided to reproduce the reported results while capturing the subtleties of instruction following.
Dataset Structure
Each sample in the dataset follows this structured format:
Example Entry
{ "key": 4767, "prompt": "فسر كيف يمكن للتقنيات الحديثة مثل الذكاء الاصطناعي أن تسهم في الحفاظ على الأدب العربي، مع تضمين 12 كلمة تنتهي بأحد الحروف الرافسة (د، ذ، أ، ر، ز، و) ، وأن تكون الإجابة مكتوبة بأسلوب موجز لا يتجاوز 120 كلمة. يجب أن لا تحتوي اجابتك على أي فواصل.", "instruction_id_list": [ "keywords:letter_list_freq", "punctuation:no_comma", "length_constraints:number_words" ], "kwargs": "[{\"letters\": [\"د\", \"ذ\", \"أ\", \"ر\", \"ز\", \"و\"], \"frequency\": 12, \"relation\": \"at least\", \"position\": \"end\"}, {}, {\"relation\": \"less than\", \"num_words\": 500}]", "lang": ["ar"] }
Field Descriptions
key: Unique identifier for each sample.
prompt: The instruction given to the model, including linguistic and formatting constraints.
instruction_id_list: Categories of constraints applied to the prompt, such as punctuation restrictions or word limits.
kwargs: JSON-encoded constraints defining specific response requirements.
lang: Specifies the allowed response languages.