dataset_info:
features:

• name: key
  dtype: int64
• name: prompt
  dtype: string
• name: instruction_id_list
  list: string
• name: kwargs
  list:
  • name: num_highlights
    dtype: int64
  • name: relation
    dtype: string
  • name: num_words
    dtype: int64
  • name: num_placeholders
    dtype: int64
  • name: prompt_to_repeat
    dtype: string
  • name: num_bullets
    dtype: int64
  • name: section_spliter
    dtype: string
  • name: num_sections
    dtype: int64
  • name: capital_relation
    dtype: string
  • name: capital_frequency
    dtype: int64
  • name: keywords
    list: string
  • name: num_paragraphs
    dtype: int64
  • name: language
    dtype: string
  • name: let_relation
    dtype: string
  • name: letter
    dtype: string
  • name: let_frequency
    dtype: int64
  • name: end_phrase
    dtype: string
  • name: forbidden_words
    list: string
  • name: keyword
    dtype: string
  • name: frequency
    dtype: int64
  • name: num_sentences
    dtype: int64
  • name: postscript_marker
    dtype: string
  • name: first_word
    dtype: string
  • name: nth_paragraph
    dtype: int64
    splits:
• name: test
  num_bytes: 311781
  num_examples: 541
  download_size: 116398
  dataset_size: 311781
  configs:
• config_name: default
  data_files:
  • split: test
    path: data/test-*
    license: apache-2.0
    task_categories:
• text-generation
  language:
• et
  size_categories:
• n<1K

Estonian Instruction-Following Eval (IFEval-et)

Description

ifeval_etis an Estonian adaptation of the original IFEval (Zhou et al., 2023) designed to evaluate instruction-following capabilities of language models in Estonian.
The dataset tests whether models can follow specific formatting, content, and linguistic constraints while generating responses. The adaptation was performed by hand by a professional translator.

The following key modifications were made:

• All content translated to Estonian following Zhou et al. 2023 guidelines
• Maintained linguistic variability with multiple ways to express identical instructions
• Cultural references localized (e.g., "President of the United States" → "President of Estonia")
• Geographic and historical references adapted to Estonian context
• Core themes preserved while using Estonian-specific content

Model Performance

The following results were obtained with the LM Evaluation Harness in zero-shot mode with temperature set to 0.

Proprietary

Open Weights

Using with the Language Model Evaluation Harness

The dataset is designed as a drop-in replacement for the English version. To prepare for the evaluation, simply place the following yaml file in the same folder as the original dataset:

task: ifeval-et


dataset_path: tartuNLP/ifeval_et
dataset_name: null

output_type: generate_until
test_split: test
num_fewshot: 0
doc_to_text: prompt
doc_to_target: 0
generation_kwargs:
  until: []
  do_sample: false
  temperature: 0.0
  max_gen_toks: 1280
process_results: !function utils.process_results
metric_list:
  - metric: prompt_level_strict_acc
    aggregation: mean
    higher_is_better: true
  - metric: inst_level_strict_acc
    aggregation: !function utils.agg_inst_level_acc
    higher_is_better: true
  - metric: prompt_level_loose_acc
    aggregation: mean
    higher_is_better: true
  - metric: inst_level_loose_acc
    aggregation: !function utils.agg_inst_level_acc
    higher_is_better: true
metadata:
  version: 4.0

Changelog

• 30.12.2025 Corrected the erroneus constraints for the example with key 2685. The original uncorrected dataset is still accessible via the original-v1 branch.

Citation

TBA