dataset_info:
features:

• name: key
  dtype: int64
• name: prompt
  dtype: string
• name: instruction_id_list
  sequence: string
• name: kwargs
  list:
  • name: count
    dtype: int64
  • name: end_phrase
    dtype: string
  • name: ending
    dtype: string
  • name: first_word
    dtype: string
  • name: forbidden_words
    sequence: string
  • name: frequency
    dtype: int64
  • name: kanji_limit
    dtype: int64
  • name: keyword
    dtype: string
  • name: keywords
    sequence: string
  • name: language
    dtype: string
  • name: let_frequency
    dtype: int64
  • name: let_relation
    dtype: string
  • name: letter
    dtype: string
  • name: nth_paragraph
    dtype: int64
  • name: num_bullets
    dtype: int64
  • name: num_highlights
    dtype: int64
  • name: num_items
    dtype: int64
  • name: num_letters
    dtype: int64
  • name: num_paragraphs
    dtype: int64
  • name: num_placeholders
    dtype: int64
  • name: num_sections
    dtype: int64
  • name: num_sentences
    dtype: int64
  • name: postscript_marker
    dtype: string
  • name: prompt_to_repeat
    dtype: string
  • name: relation
    dtype: string
  • name: section_spliter
    dtype: string
    splits:
• name: train
  num_bytes: 88795
  num_examples: 172
  download_size: 37038
  dataset_size: 88795
  configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*

JIF-Eval: Japanese Instruction Following Evaluation

Dataset Description

This dataset contains Japanese instruction-following evaluation tasks designed to test language models' ability to follow specific instructions in Japanese. It is based on the IFEval framework but adapted for Japanese language evaluation.

Dataset Details

• Total Samples: {len(dataset):,}
• Language: Japanese
• Format: Instruction-following tasks with constraints
• Purpose: Evaluation of instruction-following capabilities in Japanese

Instruction Types Distribution

The dataset includes the following types of instruction constraints:

{chr(10).join([f"- {instruction}: {count} samples" for instruction, count in instruction_counts.most_common(20)])}

{f"... and {len(instruction_counts) - 20} more instruction types" if len(instruction_counts) > 20 else ""}

Data Fields

• key: Unique identifier for each sample
• prompt: The instruction prompt in Japanese
• instruction_id_list: List of instruction constraint IDs that apply to this sample
• kwargs: Parameters for the instruction constraints (e.g., required keywords, format constraints)

Instruction Constraint Categories

The constraints are organized into several categories:

1. Language Constraints (ja:language:*): Requiring specific languages in responses
2. Length Constraints (ja:length_constraints:*): Controlling response length in sentences, characters, or paragraphs
3. Format Constraints (ja:detectable_format:*): Requiring specific formats like JSON, bullet points, numbered lists
4. Content Constraints (ja:detectable_content:*): Requiring specific content like placeholders or postscripts
5. Keyword Constraints (ja:keywords:*): Controlling keyword usage, frequency, or forbidding certain words
6. Character/Letter Constraints (ja:letters:*): Controlling use of hiragana, katakana, kanji, or furigana
7. Punctuation Constraints (ja:punctuation:*): Controlling punctuation usage
8. Start/End Constraints (ja:startend:*): Requiring specific beginnings or endings
9. Combination Constraints (ja:combination:*): Complex multi-part instructions

Usage

This dataset is designed for:

1. Model Evaluation: Test Japanese language models' instruction-following capabilities
2. Benchmarking: Compare models on Japanese instruction adherence
3. Research: Analyze instruction-following patterns and failure modes
4. Fine-tuning: Training models to better follow Japanese instructions

Example Usage

from datasets import load_dataset

dataset = load_dataset("{self.org_name}/{self.dataset_name}")

# Get a sample
sample = dataset['train'][0]
print(f"Key: {{sample['key']}}")
print(f"Prompt: {{sample['prompt']}}")
print(f"Instructions: {{sample['instruction_id_list']}}")
print(f"Parameters: {{sample['kwargs']}}")

Evaluation

To evaluate a model on this dataset:

1. Generate responses for each prompt
2. Check if the response satisfies all constraints listed in instruction_id_list
3. Use the parameters in kwargs to validate constraint compliance
4. Calculate the percentage of instructions correctly followed

Citation

If you use this dataset, please cite:
https://github.com/lightblue-tech/M-IFEval/blob/main/data/ja_input_data.jsonl

Disclaimer

This dataset contains instruction-following tasks in Japanese. It is intended for research and evaluation purposes only.