JJS Eval Results
Benchmark Results Summary
| Benchmark | Metric | Score |
|---|---|---|
| GSM8K | Pass@1 (symbolic_correct) | 80.97% |
| Hendrycks Math | Pass@1 (symbolic_correct) | 63.78% |
| Minerva Math | Pass@1 (symbolic_correct) | 25.74% |
| Human-Eval | Pass@1 (base / plus) | 59.15% / 53.05% |
| MBPP | Pass@1 (base / plus) | 68.78% / 55.82% |
| IFEval | Average Score | 56.29% |
| IFBench | Average Score | 15.96% |
Detailed Results
Math Benchmarks
| GSM8K | Hendrycks Math | Minerva Math | |
|---|---|---|---|
| Entries | 1,319 | 5,000 | 272 |
| Pass@1 | 80.97% | 63.78% | 25.74% |
| No Answer Rate | 0.76% | 4.58% | 6.99% |
| Avg Tokens | 310 | 608 | 566 |
Code Benchmarks
| Human-Eval | MBPP | |
|---|---|---|
| Entries | 164 | 378 |
| Pass@1 (base) | 59.15% | 68.78% |
| Pass@1 (plus) | 53.05% | 55.82% |
| Avg Tokens | 179 | 60 |
Instruction Following Benchmarks
| IFEval | IFBench | |
|---|---|---|
| Prompts | 541 | 294 |
| Instructions | 834 | 335 |
| Prompt Strict Acc | 49.91% | 12.93% |
| Instruction Strict Acc | 60.31% | 15.52% |
| Instruction Loose Acc | 62.83% | 19.40% |
| Average Score | 56.29% | 15.96% |
File Structure
Each benchmark directory contains:
metrics.json- Summary metricsoutput.jsonl- Detailed generation outputs