Metraipolis › archive › dc_cd92f53b2c8f

Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning — arXiv:2609.32235

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.32235
License (as archived)
CC0-1.0
Archived
2026-10-07T21:50:02.225Z
SHA-256 of the archived text
ab2f86a3844ffb322c77f09756db1dc5079d19f2f9e70f27720531c4107e1dbd
Stable URL of this page
https://metraipolis.com/documento/dc_cd92f53b2c8f

How to cite

Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning — arXiv:2609.32235 — arXiv (en), https://arxiv.org/abs/2609.32235 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cd92f53b2c8f (Metraipolis archive dc_cd92f53b2c8f, sha256 ab2f86a3844ffb322c77f09756db1dc5079d19f2f9e70f27720531c4107e1dbd)
BibTeX
@misc{dccd92f53b2c8f,
  title = {Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning — arXiv:2609.32235},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32235},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cd92f53b2c8f; sha256 ab2f86a3844ffb322c77f09756db1dc5079d19f2f9e70f27720531c4107e1dbd},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32235. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) can solve every intermediate step of a multi-step math problem on its own and still fail the full problem, even when given a roadmap of the steps and all of their answers. We introduce OracleLadder, a diagnostic evaluation that locates where LLM math reasoning fails by giving the model increasing levels of oracle help. For each problem, a teacher model writes a fixed roadmap of intermediate sub-goals (milestones), and a deterministic symbolic verifier grades every answer. Testing the model with no help, with the roadmap, with the roadmap plus the milestone answers, and on each milestone alone sorts each failure into one of five reasoning gaps. On 354 NuminaMath problems and six models from 8B to 671B parameters (Qwen3, gpt-oss, Llama 3.3, DeepSeek-V3.1), the largest gap for every model is the composition gap, a stricter form of the compositionality gap. It covers 33-48% of problems, and 24-37% after removing problems that an LLM review flags as grading errors. Accuracy and milestone-help recovery rank the two strongest models differently, and two RLVR runs with similar accuracy gains move problems differently. The roadmap effect replicates on MATH500 and AIME 2024/25, per-problem recovery agrees for 83-87% of problems under an independent second teacher, and the help ladder carries over to code generation. We release the data, roadmaps, prompts, and code at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.32235 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T21:27:08.406Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).