license: apache-2.0
task_categories:
- text-generation
language: - en
tags: - instruction-following
- ifeval
- evaluation
- verifiable-rewards
size_categories: - n<1K
configs: - config_name: default
data_files:- split: test
path: data/test.jsonl
- split: test
IFEval-mini-120
A deterministic 120-prompt subset of google/IFEval
that still covers all 25 instruction types. Built for evaluating small and
compute-constrained models — especially temporally-gated / historical LMs, where a full 541-prompt
sweep across many model variants is expensive.
Scoring is unchanged from IFEval, so per-category numbers remain interpretable; only the sample
size differs.
Composition
| prompts | 120 |
| instruction types covered | 25 / 25 |
| single-constraint prompts | 69 |
| two-constraint prompts | 40 |
| three-constraint prompts | 11 |
| total instruction instances | 182 |
Selection is random.Random(0).shuffle(...) over the 541 IFEval rows, then the first 120 — fully
reproducible via build_subset.py. Fields are byte-identical to the source rows
(key, prompt, instruction_id_list, kwargs).
Usage
lm-evaluation-harness (scoring delegates to the harness's own IFEval implementation, so numbers
are directly comparable to published IFEval results):
lm_eval --model hf --model_args pretrained=<model> \
--tasks ifeval_mini120 --include_path ./ --output_path out/
Directly:
from datasets import load_dataset
ds = load_dataset("json", data_files={"test": "data/test.jsonl"})["test"]
Reference results
Measured on this exact subset — greedy decoding, seed 0, strict scoring. Useful as a sanity check
that your harness is wired up correctly.
| model | params | prompt-strict | inst-strict |
|---|---|---|---|
SmolLM3-3B (/no_think) |
3B | 69.2 | 76.9 |
| Mistral-7B-Instruct-v0.2 | 7B | 49.2 | 59.3 |
| typewriter-1938-7B-instruct | 7B | 17.5 | 27.5 |
| typewriter-1913-7B-sft-v2 | 7B | 15.8 | 23.6 |
| typewriter-1913-7B-dpo-v2 | 7B | 14.2 | 21.4 |
| typewriter-1913-7B-base-v2 | 7B | 14.2 | 23.6 |
| violet-1b4-chat | 1.4B | 10.8 | 18.1 |
| talkie-1930-13b-it (nf4) | 13B | 8.3 | 19.2 |
Scored through this task definition (seeded, deterministic). Talkie was run at
bitsandbytes nf4 and batch=1; all others bf16 or fp8 via vLLM.
Caveats
- n=120 is small. The 95% Wilson interval on prompt-level accuracy is roughly ±7–9pp. Use
this for iteration and for screening many variants; quote full IFEval for headline claims. - Aggregate scores flatter base models. An untuned model that rambles satisfies length- and
keyword-style constraints by accident, and can satisfy format constraints by echoing the prompt
— IFEval's format prompts contain literal<<title>>examples. Report per-category, and consider
reporting prompt/response n-gram overlap alongside any base-model score. - Left-padded batching can silently corrupt some architectures. Verify your generations are not
degenerate before trusting a low score. - Upstream IFEval scoring is nondeterministic; this task fixes it. lm-eval-harness calls
langdetectwithout seeding it, solanguage:response_languagecan flip between runs — we
measured the same generations scoring 49.17 / 50.00 / 49.17 on three identical passes.utils.py
here setsDetectorFactory.seed = 0, making repeated scoring reproducible. If you compare models
within ~1pp on stock IFEval, some of that difference is noise.
License & attribution
Apache-2.0, inherited from google/IFEval
(Zhou et al., 2023, Instruction-Following Evaluation for Large Language Models). This is a
redistributed subset with no modification to prompts or scoring.
@article{zhou2023instructionfollowing,
title={Instruction-Following Evaluation for Large Language Models},
author={Zhou, Jeffrey and Lu, Tianjian and Mishra, Swaroop and Brahma, Siddhartha
and Basu, Sujoy and Luan, Yi and Zhou, Denny and Hou, Le},
journal={arXiv preprint arXiv:2311.07911},
year={2023}
}