base_model: allenai/Olmo-3-7B-Think
library_name: peft
license: apache-2.0
language:
- en
pipeline_tag: text-generation
datasets: - HuggingFaceH4/MATH-500
tags: - peft
- lora
- sft
- chain-of-thought
- reasoning
- compression
- math
model-index: - name: cot-dialect-math-olmo3-7b-think-sft-unfiltered-l5
results:- task:
type: text-generation
name: Mathematical Reasoning
dataset:
name: MATH-500
type: HuggingFaceH4/MATH-500
split: test
metrics:- type: accuracy
value: 63.2
name: Accuracy (exact match)
- type: accuracy
- task:
Olmo-3-7B-Think — L5 dialect (Pure expression) · MATH
A LoRA adapter that makes allenai/Olmo-3-7B-Think reason at compression level L5 — a single collapsed expression.
Results
| Accuracy | |
|---|---|
| This adapter | 63.2% |
MATH-500 (n=500), greedy decoding, single-turn, no exemplars, no self-consistency.
Scored with the project's LaTeX-aware grader (see the scoring note below).
Scoring note. MATH answers are
\boxed{}, and the harness that produced the first pass of these evals looked for GSM8K's#### n. That silently scored three of these models at ~0%% when they were near 60%%. Numbers here come from the project's LaTeX-aware grader, which normalizes equivalent forms (\frac{14}{3}==14/3).
Training data
MATH training problems re-expressed at level L5 by a teacher model. MATH ships three levels rather than five — L1 anchor, L3 symbolic middle, L5 extreme — with the notation rules held identical to the GSM8K dialects and only the answer convention changed to \boxed{}.
This is the unfiltered corpus.
Training setup
| Stage | supervised fine-tuning (distillation) |
| Engine | HuggingFace transformers + peft |
| LoRA | r=16, alpha=32, dropout=0.05 |
| Epochs | 3 |
| Learning rate | 2e-4, cosine, warmup 0.03 |
| Batch | 16 x 4 grad-accum = 64 effective |
| Max sequence | 1024 |
| Precision | bf16 |
| Hardware | 1x NVIDIA A100 80GB |
Loss is on the completion only, with prompt lengths precomputed at load time rather than found by pattern search — the pattern-search collator silently masked nothing, which let the base model's tool-calling prior leak into the chains.
Usage
Solve this using Level 5 (Extreme).
Problem: {your problem}
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
model = AutoModelForCausalLM.from_pretrained("allenai/Olmo-3-7B-Think", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "ssurface/cot-dialect-math-olmo3-7b-think-sft-unfiltered-l5")
tok = AutoTokenizer.from_pretrained("allenai/Olmo-3-7B-Think")
Limitations
- Trained and evaluated on math word problems only.
- Accuracy falls with problem difficulty, fastest at the compressed levels.
- Single seed unless the repo name says otherwise; differences of a couple of points are within noise (95% half-width ~2.7 pp at n=1317, ~4.4 pp at n=500).
Citation
@misc{cot-compression-dialects,
title = {Chain-of-Thought Compression Dialects},
author = {Frolov, Anatolii},
year = {2026}
}