pretty_name: DeviceMark battery, raw runs (Teraformer evalboard)
language:

• en
  tags:
• evaluation
• benchmark
• on-device
• ifeval
• mmlu-pro
• math
  viewer: false

DeviceMark battery: raw runs from Teraformer's evaluation board

Every answer behind the DeviceMark-protocol scores on our board. One folder holds one row of the board: each prompt as it was sent, the model's reply and its thinking, the parsed answer, whether it is right, and the tokens it used. We publish them so that any score can be checked.

These are our own runs of DeviceMark's protocol on our hardware. They are not DeviceMark's results, and DeviceMark has not reviewed them. Theirs are at devicemark/results.

The rows

Scores as of 4 Oct 2026. Each folder's scores.json is the record.

• Composite: the mean of the three tests.
• Answered: the share of items that got an answer within the token cap. An item with no answer counts as wrong.
• Thinking: "none" is a model that has no thinking mode. "always" is a model that can't switch it off.

The protocol (devicemark-replica-v1, scoring v2)

• 0-shot. The item's prompt is the one user message, in the model's own chat template.
• Greedy: temperature 0, seed 0.
• A cap of 4,096 generated tokens, thinking included.
• Thinking off: the whole reply is scored.
• Thinking on or always: only the answer after the thinking closes is scored. A reply that reaches the cap while still thinking has no answer. This is why a small model can score far lower with thinking on: it spends the whole cap thinking.
• Run with Hugging Face transformers through lm_eval, in bfloat16, on one RTX 5090. Each folder's setup.json says where its row ran.

The battery

The IFEval items are the same 300 that DeviceMark uses. The MMLU-Pro and MATH items are our own seeded draw of DeviceMark's design, so those two scores follow the same design on different items.

How a score is computed

• IFEval: its official checkers. The score is the mean of prompt-level and instruction-level accuracy, strict and loose.
• MMLU-Pro: the letter in the last \boxed{}, or else one of a short list of unambiguous phrasings (parsed_how says which).
• MATH: the last \boxed{}, equal as maths (math-verify).
• Intervals: Wilson's 95% for each test. The composite's is an item bootstrap: 2,000 resamples within each test, seed 0.

Check a score yourself

hf download Teraformer-limited/evalboard-devicemark-raw --repo-type dataset \
  --include "google__gemma-4-E2B-it__thinking/*" --local-dir .
python google__gemma-4-E2B-it__thinking/recompute.py

recompute.py reads items.jsonl, computes scores.json again from each item's verdict, and says whether the two agree. It needs only Python's standard library.

In each folder

• items.jsonl: one item a line. DeviceMark's field names where ours mean the same (key, answer, capped, generated_tokens, prompt_chars, content_chars, thinking_chars), and ours beside them (test, prompt, thinking, answered, correct, parsed, parsed_how, gold, ifeval, rules).
• scores.json: each test's score and interval, and the composite.
• setup.json: the run's setup.
• log.txt: the run's log, with keys, paths, host names and private addresses removed.
• recompute.py and a README.md for that row.

The prompts belong to their source datasets, under the licences above. The replies are the models' own output.