size_categories: n<1K
dataset_info:
features:
- name: instruction
dtype: string - name: response
dtype: string - name: model_name
dtype: string - name: instruction_id_list
sequence: string - name: distilabel_metadata
struct:- name: raw_input_i_f_eval_kwargs_assignator_0
list:- name: content
dtype: string - name: role
dtype: string
- name: content
- name: raw_output_i_f_eval_kwargs_assignator_0
dtype: string
- name: raw_input_i_f_eval_kwargs_assignator_0
- name: kwargs
dtype: string
splits: - name: train
num_bytes: 4280362
num_examples: 500
download_size: 490471
dataset_size: 4280362
configs: - config_name: default
data_files:- split: train
path: data/train-*
tags:
- split: train
- synthetic
- distilabel
- rlaif
Dataset Card for ifeval-like-data
This dataset has been created with distilabel.
Dataset Summary
This dataset contains a pipeline.yaml which can be used to reproduce the pipeline that generated it in distilabel using the distilabel CLI:
distilabel pipeline run --config "https://huggingface.co/datasets/gabrielmbmb/ifeval-like-data/raw/main/pipeline.yaml"
or explore the configuration:
distilabel pipeline info --config "https://huggingface.co/datasets/gabrielmbmb/ifeval-like-data/raw/main/pipeline.yaml"
Dataset structure
The examples have the following structure per configuration:
Configuration: default
{
"distilabel_metadata": {
"raw_input_i_f_eval_kwargs_assignator_0": [
{
"content": "You will receive a list of constraints and an instruction. The instruction contains constraints. Your task is to generate the appropriate arguments for each constraint type.\n\n## Constraint types, descriptions, and arguments\n\n1. keywords:existence - Checks if specific keywords are present in the response.\n - keywords: List[str]\n\n2. keywords:frequency - Verifies if a keyword appears a certain number of times in the response.\n - keyword: str\n - frequency: int\n - relation: str\n\n3. keywords:forbidden_words - Ensures that specified words are not used in the response.\n - forbidden_words: List[str]\n\n4. keywords:letter_frequency - Checks if a particular letter appears a certain number of times in the response.\n - letter: str (single letter)\n - let_frequency: int\n - let_relation: str\n\n5. language:response_language - Verifies that the entire response is in a specified language.\n - language: str (ISO 639-1 language code)\n\n6. length_constraints:number_sentences - Checks if the response contains a specific number of sentences.\n - num_sentences: int\n - relation: str\n\n7. length_constraints:number_paragraphs - Verifies that the response has a particular number of paragraphs.\n - num_paragraphs: int\n\n8. length_constraints:number_words - Ensures the response contains a specified number of words.\n - num_words: int\n - relation: str\n\n9. length_constraints:nth_paragraph_first_word - Checks if a specific paragraph starts with a particular word.\n - num_paragraphs: int\n - nth_paragraph: int\n - first_word: str\n\n10. detectable_content:number_placeholders - Verifies that the response includes a certain number of placeholders (e.g., [placeholder]).\n - num_placeholders: int\n\n11. detectable_content:postscript - Checks if the response includes a postscript section.\n - postscript_marker: str\n\n12. detectable_format:number_bullet_lists - Ensures the response contains a specific number of bullet point lists.\n - num_bullets: int\n\n13. detectable_format:constrained_response - Verifies that the response matches one of a set of predefined options.\n - (No additional arguments required)\n\n14. detectable_format:number_highlighted_sections - Checks if the response includes a certain number of highlighted sections.\n - num_highlights: int\n\n15. detectable_format:multiple_sections - Ensures the response is divided into a specified number of sections.\n - section_spliter: str\n - num_sections: int\n\n16. detectable_format:json_format - Verifies that the entire response is in valid JSON format.\n - (No additional arguments required)\n\n17. detectable_format:title - Checks if the response includes a title wrapped in double angular brackets.\n - (No additional arguments required)\n\n18. combination:two_responses - Ensures that two distinct responses are provided, separated by asterisks.\n - (No additional arguments required)\n\n19. combination:repeat_prompt - Verifies that the original prompt is repeated before the answer is given.\n - prompt_to_repeat: str\n\n20. startend:end_checker - Checks if the response ends with a specific phrase.\n - end_phrase: str\n\n21. change_case:capital_word_frequency - Verifies that a certain number of words are in all capital letters.\n - capital_frequency: int\n - capital_relation: str\n\n22. change_case:english_capital - Ensures the entire response is in English and uses all capital letters.\n - (No additional arguments required)\n\n23. change_case:english_lowercase - Checks that the entire response is in English and uses all lowercase letters.\n - (No additional arguments required)\n\n24. punctuation:no_comma - Verifies that the response does not contain any commas.\n - (No additional arguments required)\n\n25. startend:quotation - Ensures the entire response is wrapped in double quotation marks.\n - (No additional arguments required)\n\nAll the arguments are optional.\n\n## Instructions\n\n1. Analyze the provided list of constraints and the given instruction carefully.\n2. For each constraint in the list, identify the relevant parameters from the instruction text.\n3. If a constraint type is not in the list of constraints then all its arguments should be `null`.\n4. Use appropriate data types for the kwargs (strings, integers, booleans, lists, etc.).\n5. If an argument is not relevant, then its value must be `null`.\n6. Be precise and avoid adding unnecessary or speculative kwargs.\n7. For `*_relation` arguments, only provide a non-null value if the corresponding main argument is also non-null.\n\n## Output format:\n\n```\n{{\n \"keywords\": [...],\n \"keyword\": ...,\n \"frequency\": ...,\n \"relation\": ...,\n \"forbidden_words\": [...],\n \"letter\": ...,\n \"let_frequency\": ...,\n \"let_relation\": ...,\n \"language\": \"...\",\n \"num_sentences\": ...,\n \"num_paragraphs\": ...,\n \"num_words\": ...,\n \"nth_paragraph\": ...,\n \"first_word\": ...,\n \"num_placeholders\": ...,\n \"postscript_marker\": ...,\n \"num_bullets\": ...,\n \"num_highlights\": ...,\n \"section_spliter\": ...,\n \"num_sections\": ...,\n \"prompt_to_repeat\": ...,\n \"end_phrase\": ...,\n \"capital_frequency\": ...,\n \"capital_relation\": ...\n}}\n```\n",
"role": "system"
},
{
"content": "## Instruction\n\nYour response should contain 4 sentences. Your ENTIRE response should be in SPANISH language, no other language is allowed. Include keywords luna, mar, cielo, and viento. Finish your response with this exact phrase \"hasta siempre\". No other words should follow this phrase.\n## Instruction ID List\n\nkeywords:existence\nlanguage:response_language\nlength_constraints:number_sentences\nstartend:end_checker",
"role": "user"
}
],
"raw_output_i_f_eval_kwargs_assignator_0": "{ \"language\": \"es\", \"num_sentences\": 4, \"relation\": \"at least\", \"num_placeholders\": null, \"num_bullets\": null, \"starter\": null, \"num_highlights\": null, \"section_spliter\": \"SECTION\", \"num_sections\": null, \"num_paragraphs\": null, \"postscript_marker\": \"P.S.\", \"original_message\": null, \"keywords\": [\"luna\", \"mar\", \"cielo\", \"viento\"], \"keyword\": null, \"frequency\": null, \"num_words\": null, \"nth_paragraph\": null, \"first_word\": null, \"key_sentences\": null, \"forbidden_words\": null, \"original_paragraph\": null, \"low\": null, \"high\": null, \"prompt_to_repeat\": null, \"end_phrase\": \"hasta siempre\", \"letter\": null, \"let_frequency\": null, \"let_relation\": \"at least\", \"capital_frequency\": null, \"capital_relation\": \"at least\" }"
},
"instruction": "Your response should contain 4 sentences. Your ENTIRE response should be in SPANISH language, no other language is allowed. Include keywords luna, mar, cielo, and viento. Finish your response with this exact phrase \"hasta siempre\". No other words should follow this phrase.\n",
"instruction_id_list": [
"keywords:existence",
"language:response_language",
"length_constraints:number_sentences",
"startend:end_checker"
],
"kwargs": "{ \"language\": \"es\", \"num_sentences\": 4, \"relation\": \"at least\", \"num_placeholders\": null, \"num_bullets\": null, \"starter\": null, \"num_highlights\": null, \"section_spliter\": \"SECTION\", \"num_sections\": null, \"num_paragraphs\": null, \"postscript_marker\": \"P.S.\", \"original_message\": null, \"keywords\": [\"luna\", \"mar\", \"cielo\", \"viento\"], \"keyword\": null, \"frequency\": null, \"num_words\": null, \"nth_paragraph\": null, \"first_word\": null, \"key_sentences\": null, \"forbidden_words\": null, \"original_paragraph\": null, \"low\": null, \"high\": null, \"prompt_to_repeat\": null, \"end_phrase\": \"hasta siempre\", \"letter\": null, \"let_frequency\": null, \"let_relation\": \"at least\", \"capital_frequency\": null, \"capital_relation\": \"at least\" }",
"model_name": "Qwen/Qwen2.5-72B-Instruct",
"response": "La luna se refleja en el mar mientras el cielo se oscurece y el viento susurra suavemente. Cada ola parece contener un suspiro del viento que acaricia la orilla. El cielo, lleno de estrellas, contrasta con la oscuridad del mar bajo la luz de la luna. El viento contin\u00faa su melod\u00eda nocturna, uniendo el cielo y el mar en un espect\u00e1culo eterno hasta siempre."
}
This subset can be loaded as:
from datasets import load_dataset
ds = load_dataset("gabrielmbmb/ifeval-like-data", "default")
Or simply as it follows, since there's only one configuration and is named default:
from datasets import load_dataset
ds = load_dataset("gabrielmbmb/ifeval-like-data")
References
@misc{xu2024magpiealignmentdatasynthesis,
title={Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing},
author={Zhangchen Xu and Fengqing Jiang and Luyao Niu and Yuntian Deng and Radha Poovendran and Yejin Choi and Bill Yuchen Lin},
year={2024},
eprint={2406.08464},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2406.08464},
}