license: apache-2.0
language:

• en
  pretty_name: needleif-bench
  tags:
• long-context
• instruction-following
• ifeval
• needle-in-a-haystack
• evaluation
• catastrophic-forgetting
  task_categories:
• text-generation
  configs:
• config_name: default
  data_files:
  • split: test
    path: data/full/test.jsonl
  • split: train
    path: data/full/train.jsonl
  • split: val
    path: data/full/val.jsonl

needleif-bench

A judge-free, NIAH-style long-context control benchmark with hidden verifiable
instructions. The "needle" (lifted from IFEval)
is hidden inside long distractor prose (the "haystack"). The model must find the
instruction and follow it — testing long-context retrieval and instruction
following at once, with deterministic, model-free scoring.

It exists to measure capability regression (catastrophic forgetting) after
task-specific fine-tuning: run it on a model before and after unrelated FT and
watch the score move. Most useful for small open-weight models.

from datasets import load_dataset
ds = load_dataset("lefft/needleif-bench", split="test")  # "test" | "train" | "val"
ds_128k = ds.filter(lambda r: r["metadata"]["size_bucket"] == "128k")

Task

Every needle is presented in two conditions that differ only in the
presence of the document, so the haystack is the sole variable:

• bare — the instruction alone: "Follow it. Reply with ONLY the output…"
• haystack — the same instruction buried in long prose: "Hidden somewhere in
  this document is exactly one instruction. Find it and follow it. Reply with
  ONLY the output…"

Metrics

A response passes a needle when it satisfies all of that needle's
instructions (prompt-level). Each pass is scored two ways: strict (exact)
and loose (forgives a stray preamble line or stray markdown); unless noted,
tables use loose prompt-level pass@1 — one sample at temperature 0. pass@k is
also supported.

From each bare/haystack pair we compute:

• baseline (bare) accuracy — fraction of needles the model follows given the
  instruction alone.
• long-context (haystack) accuracy — fraction it still follows when the
  instruction is buried in the haystack. (This is the "long-context loose
  prompt@1" column in the tables.)
• distraction regression = #(passes bare ∧ fails haystack) / #(passes bare) —
  of the needles the model can follow in isolation, the fraction the haystack
  causes it to fail. This is the primary number: it divides out raw
  instruction-following ability to isolate the long-context effect.
• net penalty = baseline − long-context accuracy.

All are broken down by haystack size, needle depth, and instruction complexity.

Results

Stale: the tables below were measured on the pre-2026-07-10 data
(cl100k-sized buckets, no per-bucket budgets). Rankings are unlikely to
change materially, but rerun run_model.py + score.py to refresh.

A few models on the full test split (100 pairs) — a small illustrative set, not a leaderboard; loose prompt-level pass@1, seed=666.

Long-context (haystack) accuracy by size:

Dataset statistics

300 needles → 600 records across three splits with disjoint needle pools (no
leakage across splits), seed=666. Per-split, per-size needle (pair) counts —
smaller sizes are sampled less so the long-context tail stays well-represented:

Each needle also appears in a bare condition (the instruction alone). Train
and val records carry a verified assistant turn (see Record schema below);
test records are user-only. Each size
bucket is a context window: the chat-templated conversation — prompt plus
the assistant turn (train/val) or the reserved generation budget (test, =
output_reserve_tokens) — fits within 95% of the bucket, as measured by the
pinned Gemma 4 tokenizer used for sizing. Training at seq_len == bucket or
evaluating a bucket-context model therefore needs no filtering. ~1/3 of needles are complex
(2–3 instructions), the rest simple (1) — bounded by the IFEval-derived pool
(209 simple / 106 complex). Depths {10%, 50%, 90%} cycle within each size bucket.
Regenerate stats with python3 scripts/dataset_stats.py --records data/full/<split>.jsonl.

Record schema

One JSON object per line. messages opens with a single user turn — a fixed
find-the-instruction meta-instruction followed by the document. Scoring never
uses a gold response: it applies the metadata spec (instruction_id_list +
kwargs) to the model's answer, and metadata is never shown to the model.

In train and val, each record additionally ends with an assistant
turn so those splits are directly usable as SFT data. Responses were generated
from the bare prompt (both records of a pair share the same response — the
checks are response-only, so the correct output is identical for both
conditions) and every one is verified to pass its needle's strict checks
(google/gemma-4-31B-it, stragglers filled by
meta-llama/Llama-3.3-70B-Instruct-Turbo; see scripts/add_responses.py).
The test split stays user-only: the eval is deterministic and needs no
reference answer.

{
  "id": "f00d…-hay",
  "pair_id": "f00d…",
  "messages": [
    {"role": "user", "content": "…fixed find-the-instruction preamble…\n\n----- BEGIN DOCUMENT -----\n…prose … the needle instruction … prose…\n----- END DOCUMENT -----"}
  ],
  "metadata": {
    "condition": "haystack",                     // or "bare"
    "instruction_id_list": ["detectable_format:title"],
    "kwargs": [{}],
    "needle_text": "the IFEval instruction, verbatim",
    "needle_source": "ifeval", "needle_source_key": 1357,
    "haystack_source": "pg19",
    "haystack_target_tokens": 8192, "haystack_actual_tokens": 5637, "size_bucket": "8k",
    "total_prompt_tokens": 5702, "prompt_token_budget": 5718, "output_reserve_tokens": 2048,
    "reference_tokenizer": "hf:google/gemma-4-E4B-it@a4c2d58be94dda072b918d9db64ee85c8ed34e3f",
    "depth_percent": 50.0, "needle_char_offset": 19342,  // offset into the document body
    "complexity": "simple", "num_instructions": 1,
    "split": "test", "seed": 666, "needle_marker": null
  }
}

Usage

See RUNBOOK.md for the full walkthrough (auth, loading via datasets, regenerating the data, reasoning models). The short version:

# get the benchmark (this repo holds both the data and the code) and install
git clone https://huggingface.co/datasets/lefft/needleif-bench && cd needleif-bench
pip install -r requirements.txt

# run a model on the test split (Anthropic, or any OpenAI-compatible endpoint)
python3 run_model.py \
    --records data/full/test.jsonl \
    --provider anthropic \
    --model claude-haiku-4-5-20251001 \
    --out responses.jsonl

# score (judge-free; responses JSONL is {"id","response"} or {"id","responses":[...]})
python3 score.py \
    --records data/full/test.jsonl \
    --responses responses.jsonl \
    --out-results results.jsonl \
    --out-summary summary.json

Scoring

We adopt a simplified, self-contained version of IFEval's scoring: each supported
instruction type has a deterministic check run against the model's answer only (no
model-as-judge). The 17 checks live in needleif/checks.py, which — like
score.py — is pure standard library, so the published dataset can be scored with
nothing installed.

Worked example — the needle "Answer with at least 3 bullet points and finish with
the exact phrase: Any other questions?" carries two checks, number_bullet_lists (≥3) and end_checker ("Any other questions?"); the answer passes the needle iff
both hold on the response text.

Notes & caveats

• Token sizes are measured in the pinned Gemma 4 tokenizer (see
  reference_tokenizer); the per-bucket fit guarantee holds exactly for that
  tokenizer and, via the 5% headroom, for tokenizers within ~5% of it on
  English prose (most recent open models). Sizing never affects pass/fail.
  Regenerate with --tokenizer hf:<model>[@rev] to retarget another family.
• output_reserve_tokens is the eval generation budget: run the test split
  with max_tokens equal to the record's reserve and the request can never
  overflow a bucket-sized window.
• Deterministic. generate.py is a pure function of its config + seed
  (byte-identical output); scoring is deterministic.
• Haystack source. Default is PG19 (Apache-2.0, redistributable). A
  zero-dependency programmatic provider exists for offline use.

Repository structure

needleif-bench/
├── needleif/               # library (the scoring path is stdlib-only)
│   ├── checks.py           # 17 IFEval checks + whitelist — the judge-free scoring core
│   ├── scoring.py          # per-record + paired distraction metrics, pass@k
│   ├── haystack.py         # PG19 + programmatic distractor providers
│   ├── sizing.py           # tokenizer abstraction (tiktoken | hf:<model> | words)
│   └── io_utils.py         # jsonl I/O, stable ids, seeded RNG
├── generate.py             # CLI: build the dataset (config, needle pool, wrappers)
├── score.py                # CLI: judge-free scoring
├── run_model.py            # CLI: run a model (Anthropic | OpenAI-compatible)
├── scripts/                # fetch_needles · add_responses · dataset_stats · results_table · push_to_hf
├── configs/                # generation grids
├── data/                   # needles_ifeval.json (needle pool) + generated splits
├── tests/                  # pytest, network-free  (run: python3 -m pytest)
├── CLAUDE.md  RUNBOOK.md   # contributor notes  ·  step-by-step eval walkthrough
├── README.md  LICENSE
└── pyproject.toml  requirements.txt

See CLAUDE.md for the module map + contributor notes, and RUNBOOK.md to run an eval end-to-end.

Related benchmarks

• LIFBench — Wu et al. (2025), ACL 2025. Instruction-following accuracy and stability under long-context inputs — the closest sibling, though it scales context around the instruction rather than hiding a single verifiable needle.
• RULER — Hsieh et al. (2024), COLM 2024. Generalizes needle-in-a-haystack with configurable needles plus multi-hop and aggregation tasks to probe a model's effective context length.
• BABILong — Kuratov et al. (2024), NeurIPS 2024 Datasets & Benchmarks. Hides bAbI reasoning facts in PG19 background text — the same task-in-PG19-noise construction used here — but targets multi-fact reasoning rather than instruction-following.
• IFEval — Zhou et al. (2023), arXiv preprint. ~500 prompts with programmatically verifiable instructions scored by deterministic code (no LLM judge); needleif-bench reuses its prompts as needles and reimplements the checks.
• MMMT-IF — Epstein et al. (2024), arXiv preprint. Disperses code-verifiable instructions across long multi-turn multimodal dialogues, scored by a judge-free Programmatic Instruction Following (PIF) metric.
• Needle in a Haystack — Kamradt (2023), open-source project. The original retrieval-at-depth pressure test (one fact in long filler across depths and lengths); needleif-bench swaps the retrievable fact for a followable instruction.

License & attribution

Apache-2.0 (see LICENSE). Built entirely from open-source artifacts:

• IFEval — instruction definitions and needle prompts (Apache-2.0):
  google-research/instruction_following_eval
  · dataset google/IFEval
• PG19 — haystack (distractor) text (Apache-2.0):
  deepmind/pg19

TODO

• Iterate on the prompt wrappers and the bare-condition framing.
• Add a needle-length filter and tune the complex-needle instruction mix.
• Handle reasoning models more gracefully (answer extraction, token-budget guidance).
• Once the format is frozen, wipe the test suite (CLAUDE.md documents how to regenerate coverage).
• (research, not practical) Fix a needle and sweep the haystack size to find
  the context length at which instruction-following breaks down — and whether that
  threshold varies by base model, shifts before/after fine-tuning, or depends on
  needle complexity.