license: apache-2.0
task_categories:

• text-generation
  language:
• tr
  pretty_name: IFEval

Dataset Card for IFEval

IFEval_tr is the Turkish localized/oriented version of IFEval dataset. It includes both human annotated/translated versions of the original dataset as well as human created entries from scratch.
The dataset do not include tasks that require model to respond in a particular language to focus on Turkish cognitive abilities.

IFEval_tr is the Turkish-oriented version of the IFEval dataset. It contains human-annotated and human-translated versions of the original dataset, as well as newly created entries (IDs > 5000). The dataset excludes tasks that require responses in a specific language, focusing solely on Turkish cognitive abilities.

Dataset Description

• Repository: COMING SOON
• Paper: https://huggingface.co/papers/2311.07911
• Point of Contact: Le Hou
• Point of Contact: Abdullah Bezir

Dataset Summary

This dataset contains 350 "verifiable instructions" based on the method proposed in the Instruction-Following Eval (IFEval) benchmark benchmark paper. It includes verifiable instructions generated through heuristic methods, such as "Write an answer longer than 400 words" or "Include the word 'artificial' at least 3 times in your response."

To load the dataset, run:

from datasets import load_dataset

ifeval = load_dataset("bezir/IFEval_tr")

Supported Tasks and Leaderboards

IFEval-tr dataset is a core test benchmark for models with conversational capabilities (trained with instructions), just like in the original dataset.

Dataset Structure

Data Instances

An example of the train split looks as follows:

{
    "key": 1000,
    "prompt": ""'https://tr.wikipedia.org/wiki/Gelibolu' Wikipedia sayfasının 300 kelime ya da daha uzun bir özetini yaz. Cevabında virgül kullanma ve en az 3 bölümü verdiğim örnek biçiminde vurgula. Örnek vurgu: *vurgulanan bölüm 1* *vurgulanan bölüm 2* *vurgulanan bölüm 3*.",
    "instruction_id_list": [
        "punctuation:no_comma",
        "detectable_format:number_highlighted_sections",
        "length_constraints:number_words",
    ],
    "kwargs": [
        {
            "num_highlights": None,
            "relation": None,
            "num_words": None,
            "num_placeholders": None,
            "prompt_to_repeat": None,
            "num_bullets": None,
            "section_spliter": None,
            "num_sections": None,
            "capital_relation": None,
            "capital_frequency": None,
            "keywords": None,
            "num_paragraphs": None,
            "language": None,
            "let_relation": None,
            "letter": None,
            "let_frequency": None,
            "end_phrase": None,
            "forbidden_words": None,
            "keyword": None,
            "frequency": None,
            "num_sentences": None,
            "postscript_marker": None,
            "first_word": None,
            "nth_paragraph": None,
        },
        {
            "num_highlights": 3,
            "relation": None,
            "num_words": None,
            "num_placeholders": None,
            "prompt_to_repeat": None,
            "num_bullets": None,
            "section_spliter": None,
            "num_sections": None,
            "capital_relation": None,
            "capital_frequency": None,
            "keywords": None,
            "num_paragraphs": None,
            "language": None,
            "let_relation": None,
            "letter": None,
            "let_frequency": None,
            "end_phrase": None,
            "forbidden_words": None,
            "keyword": None,
            "frequency": None,
            "num_sentences": None,
            "postscript_marker": None,
            "first_word": None,
            "nth_paragraph": None,
        },
        {
            "num_highlights": None,
            "relation": "at least",
            "num_words": 300,
            "num_placeholders": None,
            "prompt_to_repeat": None,
            "num_bullets": None,
            "section_spliter": None,
            "num_sections": None,
            "capital_relation": None,
            "capital_frequency": None,
            "keywords": None,
            "num_paragraphs": None,
            "language": None,
            "let_relation": None,
            "letter": None,
            "let_frequency": None,
            "end_phrase": None,
            "forbidden_words": None,
            "keyword": None,
            "frequency": None,
            "num_sentences": None,
            "postscript_marker": None,
            "first_word": None,
            "nth_paragraph": None,
        },
    ],
}

Data Fields

The data fields are as follows:

• key: A unique ID for the prompt. IDs > 5000 corresponds to entries created from scratch others may correspond to the original dataset.
• prompt: Describes the task the model should perform.
• instruction_id_list: An array of verifiable instructions. See Table 1 of the paper for the full set with their descriptions.
• kwargs: An array of arguments used to specify each verifiable instruction in instruction_id_list.

Data Splits

Testing

The testing environment was set up by updating the IFEval task within the LM Evaluation Harness repository for Turkish. The code will be open-sourced soon.
The score is the mean of inst_level_strict_acc and prompt_level_strict_acc.

IFEval-TR Leaderboard

Licensing Information

The dataset is available under the Apache 2.0 license.

Citation Information

@misc{zhou2023instructionfollowingevaluationlargelanguage,
      title={Instruction-Following Evaluation for Large Language Models}, 
      author={Jeffrey Zhou and Tianjian Lu and Swaroop Mishra and Siddhartha Brahma and Sujoy Basu and Yi Luan and Denny Zhou and Le Hou},
      year={2023},
      eprint={2311.07911},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2311.07911}, 
}
@misc{IFEval_tr,
  author = {Abdullah Bezir},
  title = {bezir/IFEval_tr},
  year = {2024},
  publisher = {Abdullah Bezir},
  howpublished = {https://huggingface.co/datasets/bezir/IFEval_tr}
}