facebook/Multi-IF

license: cc-by-nc-2.0
language:

• en
• fr
• es
• pt
• hi
• zh
• ru
• it
  tags:
• instruction following
• multi-turn
• multilingual
  size_categories:
• 1K<n<10K

Dataset Summary

We introduce Multi-IF, a new benchmark designed to assess LLMs' proficiency in following multi-turn and multilingual instructions. Multi-IF, which utilizes a hybrid framework combining LLM and human annotators, expands upon the IFEval by incorporating multi-turn sequences and translating the English prompts into another 7 languages, resulting in a dataset of 4501 multilingual conversations, where each has three turns. Our evaluation of 14 state-of-the-art LLMs on Multi-IF reveals that it presents a significantly more challenging task than existing benchmarks. All the models tested showed a higher rate of failure in executing instructions correctly with each additional turn. For example, o1-preview drops from 0.877 at the first turn to 0.707 at the third turn in terms of average accuracy over all languages. Moreover, languages with non-Latin scripts (Hindi, Russian, and Chinese) generally exhibit higher error rates, suggesting potential limitations in the models’ multilingual capabilities.

Evaluation Script

https://github.com/facebookresearch/Multi-IF

Data Fields

• turns: Placehold for saving the history conversation in evaluation.
• responses: Placehold for saving the latest response in evaluation.
• turn_1_prompt: The user prompt at the first turn, which is the input for LLM generation.
• turn_1_instruction_id_list: The instructions of the user prompt at the first turn, which is needed in the evaluation script.
• turn_1_kwargs: The arguments of the first turn instructions, which is needed in the evaluation script.
• turn_2_prompt: The user prompt at the second turn, which is the input for LLM generation.
• turn_2_instruction_id_list: The instructions of the user prompt at the second turn, which is needed in the evaluation script.
• turn_2_kwargs: The arguments of the second turn instructions, which is needed in the evaluation script.
• turn_3_prompt: The user prompt at the third turn, which is the input for LLM generation.
• turn_3_instruction_id_list: The instructions of the user prompt at the third turn, which is needed in the evaluation script.
• turn_3_kwargs: The arguments of the third turn instructions, which is needed in the evaluation script.
• key: The key of each conversation
• turn_index: Placehold for saving the current turn index in evaluation.
• language: The language of each conversation

Data Splits

• test: 4,501 examples