Metraipolis › archive › dc_48eedf60c279

Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains — arXiv:2608.03219

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.03219
License (as archived)
CC0-1.0
Archived
2026-10-31T12:15:40.130Z
SHA-256 of the archived text
31f2194b4acb7d5c682aca75dd7f5913d2822695f0908dd73b9f5c7a8c27171e
Stable URL of this page
https://metraipolis.com/documento/dc_48eedf60c279

How to cite

Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains — arXiv:2608.03219 — arXiv (en), https://arxiv.org/abs/2608.03219 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_48eedf60c279 (Metraipolis archive dc_48eedf60c279, sha256 31f2194b4acb7d5c682aca75dd7f5913d2822695f0908dd73b9f5c7a8c27171e)
BibTeX
@misc{dc48eedf60c279,
  title = {Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains — arXiv:2608.03219},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.03219},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_48eedf60c279; sha256 31f2194b4acb7d5c682aca75dd7f5913d2822695f0908dd73b9f5c7a8c27171e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03219. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Benchmark gains are often treated as evidence of greater LLM capability. Yet the same gain can reflect different changes in model behavior. A model may reach new answers, or produce answers that were already within reach. Aggregate scores do not distinguish these changes question by question. We establish a question-level audit under fixed budgets, temperatures, and answer formats. A question is realized when the default deployment procedure produces the correct answer. A question is reachable when a specified probe finds that answer within a fixed budget. We first test whether inference-time layer routing can expand reachability. Under a matched budget, random routes match or exceed structured search in all 43 model and task settings. Answer-blind procedures retain almost none of this gain, which instead requires access to the correct answer. We then ask why reachable answers sometimes fail to appear. Across six cases spanning 0.5B to 31B, silencing one identified MLP block repairs 68 to 92 percent of a predefined failure set. We next test whether training closes the gap by expanding reachability. In five of six matched evaluations, deployed performance rises while the reachable ceiling remains flat or falls. For DAPO, the deployed score rises by 14.7 points while the reachable ceiling falls by 13.3 points. Across the settings we audit, realization and reachability therefore do not always change together. Claims of capability expansion should report both realized performance and reachability under matched evaluation conditions. Code is available at this https URL

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.03219 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:05:52.768Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).