license: cc-by-4.0
language:
- en
tags: - routing
- evaluation
- llm-router
- multi-domain
- model-comparison
size_categories: - 1K<n<10K
configs: - config_name: evals
default: true
data_files:- split: coding
path: data/evals/coding.jsonl.gz - split: creative_synthesis
path: data/evals/creative_synthesis.jsonl.gz - split: instruction_following
path: data/evals/instruction_following.jsonl.gz - split: math_reasoning
path: data/evals/math_reasoning.jsonl.gz - split: planning_agentic
path: data/evals/planning_agentic.jsonl.gz - split: world_knowledge
path: data/evals/world_knowledge.jsonl.gz
- split: coding
- config_name: results
data_files:- split: coding
path: data/results/coding.jsonl.gz - split: creative_synthesis
path: data/results/creative_synthesis.jsonl.gz - split: instruction_following
path: data/results/instruction_following.jsonl.gz - split: math_reasoning
path: data/results/math_reasoning.jsonl.gz - split: planning_agentic
path: data/results/planning_agentic.jsonl.gz - split: world_knowledge
path: data/results/world_knowledge.jsonl.gz
- split: coding
- config_name: verbose
data_files:- split: coding
path: data/verbose/coding.jsonl.gz - split: creative_synthesis
path: data/verbose/creative_synthesis.jsonl.gz - split: instruction_following
path: data/verbose/instruction_following.jsonl.gz - split: math_reasoning
path: data/verbose/math_reasoning.jsonl.gz - split: planning_agentic
path: data/verbose/planning_agentic.jsonl.gz - split: world_knowledge
path: data/verbose/world_knowledge.jsonl.gz
- split: coding
Brick2 Dataset A - Routing Evaluation (regolo mirror)
Total queries: 5504 | Gated (masked): 0
Stratified routing-evaluation benchmark over 6 capability dimensions.
Each query is executed on 3 LLMs (qwen3.5-9b, deepseek-v4-flash, kimi2.6) and
graded by deterministic graders (math/coding/ifeval), LLM judge panel 2-of-3
(planning_agentic), or single judge (creative_synthesis, world_knowledge).
This repo mirrors the routing-level data published atmassaindustries/dataset-A-routing,
here re-laid as 3 configs x 6 dimension splits.
Layout
data/
evals/
coding.jsonl.gz
creative_synthesis.jsonl.gz
instruction_following.jsonl.gz
math_reasoning.jsonl.gz
planning_agentic.jsonl.gz
world_knowledge.jsonl.gz
results/
coding.jsonl.gz
creative_synthesis.jsonl.gz
instruction_following.jsonl.gz
math_reasoning.jsonl.gz
planning_agentic.jsonl.gz
world_knowledge.jsonl.gz
verbose/
coding.jsonl.gz
creative_synthesis.jsonl.gz
instruction_following.jsonl.gz
math_reasoning.jsonl.gz
planning_agentic.jsonl.gz
world_knowledge.jsonl.gz
Usage
from datasets import load_dataset
# verdict-level per modello (default config)
ds = load_dataset("regolo/brick2-dataset-a-eval", "results") # all 6 splits
ds_code = load_dataset("regolo/brick2-dataset-a-eval", "results", split="coding")
# raw responses + cost + latency + 3x3 judge per planning
ds_v = load_dataset("regolo/brick2-dataset-a-eval", "verbose", split="planning_agentic")
# prompts + ground truth (replication mode)
ds_e = load_dataset("regolo/brick2-dataset-a-eval", "evals", split="math_reasoning")
# IMPORTANT: each split file has a `_schema_anchor` row used to lock the
# pyarrow schema (nullable booleans). Filter it out before use:
ds = ds.filter(lambda r: r["query_id"] != "_schema_anchor")
Split sizes (per config, identical across configs)
| split | rows |
|---|---|
| coding | 1000 |
| creative_synthesis | 696 |
| instruction_following | 841 |
| math_reasoning | 1000 |
| planning_agentic | 1165 |
| world_knowledge | 802 |
| total | 5504 |
Note: planning_agentic_multiturn (165 rows) is merged into the planning_agentic split
to match the 6-capability framing of the paper.
Migration from v0.3
Previous revision used 7 configs (all + 6 dimensions) with single train split.
New mapping:
| v0.3 (deprecated) | v0.4 (this revision) |
|---|---|
load_dataset(REPO, "all") |
load_dataset(REPO, "evals") then concat splits |
load_dataset(REPO, "coding") |
load_dataset(REPO, "evals", split="coding") |
| (not available) | load_dataset(REPO, "results") -- verdict-level |
| (not available) | load_dataset(REPO, "verbose") -- full responses |
Schema (results)
| field | type | notes |
|---|---|---|
query_id |
string | q_NNNNN (filter _schema_anchor) |
query |
string | <masked> if gated |
dimension |
string | one of 6 capability dims |
evaluation_protocol_id |
string | grader protocol |
source |
string | original dataset |
gated |
bool | proprietary source (query masked) |
qwen_correct |
bool/null | primary verdict for qwen3.5-9b |
ds4_correct |
bool/null | primary verdict for deepseek-v4-flash |
kimi_correct |
bool/null | primary verdict for kimi2.6 |
*_correct is null when the judge abstained (~1.4% overall).
Schema (verbose)
All results fields plus per-model m in {qwen, ds4, kimi}:{m}_response, {m}_thinking, {m}_cost_usd, {m}_latency_ms, {m}_completion_tokens,{m}_reasoning_tokens, {m}_input_tokens, {m}_finish_reason, {m}_grader_meta,{m}_model_id_real, {m}_judge_gpt54mini, {m}_judge_mistral, {m}_judge_glm.
Win-rate per model
| model | correct | incorrect | abstention |
|---|---|---|---|
| qwen | 3478 | 1786 | 241 |
| ds4 | 4057 | 1220 | 228 |
| kimi | 4130 | 784 | 591 |
License
CC-BY-4.0 (except for queries from gated upstream sources, which keep their original license).
Citation
Paper: Brick - Calibrated Mixture-of-Models routing (scientificv1).
Upstream lockfile: see massaindustries/dataset-A-routing-eval.