license: cc-by-4.0
language:
- en
tags: - routing
- evaluation
- llm-router
- multi-domain
- model-comparison
size_categories: - 1K<n<10K
configs: - config_name: results
default: true
data_files:- split: train
path: data/results/train.jsonl.gz
- split: train
- config_name: verbose
data_files:- split: train
path: data/verbose/train.jsonl.gz
- split: train
- config_name: evals
data_files:- split: train
path: data/evals/train.jsonl.gz
- split: train
Dataset A - Routing (3 modelli, verdict-level)
Totale query: 5504 | Gated (masked) query: 0
Dataset di valutazione per LLM routing systems su 6 capability. Ogni query è stata eseguita su 3 modelli (qwen3.5-9b, deepseek-v4-flash, kimi2.6) e giudicata con grader deterministici (math/coding/ifeval) o LLM judge panel 2-of-3 (planning_agentic) / single judge (creative_synthesis, world_knowledge).
Configs
from datasets import load_dataset
# Pivot verdict per query (default config, 3 colonne bool)
ds_results = load_dataset("massaindustries/dataset-A-routing", split="train")
# Full: raw responses, costs, latency, 3x3 judge per planning, grader_meta
ds_verbose = load_dataset("massaindustries/dataset-A-routing", name="verbose", split="train")
# Prompts + ground truth (no model outputs) — per replicare i test
ds_evals = load_dataset("massaindustries/dataset-A-routing", name="evals", split="train")
# Tutti i config hanno una riga `_schema_anchor` (query_id == '_schema_anchor') con valori
# dummy per fissare lo schema di datasets/PyArrow. Filtrala via:
# ds = ds.filter(lambda r: r["query_id"] != "_schema_anchor")
Schema results
| campo | tipo | note |
|---|---|---|
query_id |
string | q_NNNNN |
query |
string | <masked> se gated |
dimension |
string | 1 di 6 capability |
evaluation_protocol_id |
string | protocollo grader |
source |
string | dataset originale |
gated |
bool | dataset proprietary (query mascherata) |
qwen_correct |
bool/null | verdict primario qwen3.5-9b |
ds4_correct |
bool/null | verdict primario deepseek-v4-flash |
kimi_correct |
bool/null | verdict primario kimi2.6 |
*_correct è null quando il judge ha astensione (~1.4% del totale).
Schema verbose
Tutti i campi di results + per ogni modello m in {qwen, ds4, kimi}:
{m}_response(string, raw output){m}_thinking(string, raw thinking chain){m}_cost_usd(float, null per qwen){m}_latency_ms(int){m}_completion_tokens(int){m}_reasoning_tokens(int){m}_input_tokens(int){m}_finish_reason(string){m}_grader_meta(string, JSON serializzato){m}_model_id_real(string){m}_judge_gpt54mini(bool/null, solo planning ST){m}_judge_mistral(bool/null, solo planning ST){m}_judge_glm(bool/null, solo planning ST)
Verdict source per dimension
| dimension | rows | verdict source |
|---|---|---|
| planning_agentic | 1000 | panel 2-of-3 (gpt-5.4-mini + mistral-small-2603 + glm-5-turbo) per ST; single judge per MT |
| math_reasoning | 1000 | deterministic (math_equiv, gsm8k_final_answer) |
| coding | 1000 | deterministic (unit_test_pass) |
| instruction_following | 841 | deterministic (ifeval_constraint_check) |
| world_knowledge | 802 | mix (deterministic mcq_letter per 102; LLM judge llm_judge_factual per 700) |
| creative_synthesis | 696 | LLM judge rubric_judge (gpt-5.4-mini) |
Win-rate per modello
| modello | correct | incorrect | abstention | accuracy |
|---|---|---|---|---|
| qwen | 3478 | 1786 | 241 | 0.6607 |
| ds4 | 4057 | 1220 | 228 | 0.7688 |
| kimi | 4130 | 784 | 591 | 0.8405 |
Coverage check
Ogni modello ha verdict per tutte le 5339 query:
| modello | query con verdict | coverage |
|---|---|---|
| qwen | 5339 | 97.0% |
| ds4 | 5339 | 97.0% |
| kimi | 5339 | 97.0% |
Reproducibility
- Stesso pool di 5339 query del dataset base
massaindustries/dataset-A-routing-eval - Inference deterministic (temperature=0.0)
- Judge panel costanti, prompt template versionati nello skill
llmevals - Per ripetere i test: scarica config
evals, esegui inference, applica grader
License
CC-BY-4.0 (eccetto query originali dei dataset gated, che restano sotto le license dei rispettivi source).
Citation
Per i source originali del dataset_A, vedere lockfile.yaml in massaindustries/dataset-A-routing-eval.