Metraipolis › archive › dc_685800a622ca

Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space — arXiv:2608.29188

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.29188
License (as archived): CC0-1.0
Archived: 2026-10-17T07:47:32.813Z
SHA-256 of the archived text: 5961bb87bedd663d63044f3de536d7dad4f9f2564bebe230baff18d504d34dfc
Stable URL of this page: https://metraipolis.com/documento/dc_685800a622ca

How to cite

Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space — arXiv:2608.29188 — arXiv (en), https://arxiv.org/abs/2608.29188 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_685800a622ca (Metraipolis archive dc_685800a622ca, sha256 5961bb87bedd663d63044f3de536d7dad4f9f2564bebe230baff18d504d34dfc)

BibTeX

@misc{dc685800a622ca,
  title = {Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space — arXiv:2608.29188},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.29188},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_685800a622ca; sha256 5961bb87bedd663d63044f3de536d7dad4f9f2564bebe230baff18d504d34dfc},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.29188. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning with verifiable rewards (RLVR) substantially improves single-sample accuracy (pass@1) but causes the policy's solution space to contract, diminishing the returns of test-time scaling. In this work, we investigate where inside a reasoning trajectory this breadth is lost: does the policy fail to access a valid solution family, or does it fail to execute computation once initiated? To disentangle access from execution, we analyze the Countdown task, whose solution space can be exhaustively enumerated into discrete entrance families defined by the first operand and operator, across PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct. Across both training setups, solution coverage falls by up to 67%, halving even on problems solved across all checkpoints. We show that this contraction is heavily concentrated at the entrance: per-token likelihood shifts are 11x--16x larger prior to the first arithmetic operation than during downstream reasoning. Supplying only an unselected entrance prefix restores completion rates in low-access families by over an order of magnitude (0.018 -> 0.212 under PPO), demonstrating that alternative solutions remain executable but are no longer initiated. Guided by this localization, we find that while surface prompting fails to recover diversity, entrance-targeted interventions succeed: late-layer parameter interpolation with early checkpoints increases solution coverage by 37% at no loss in pass@1. Finally, we show that early-step entropy collapse recurs across six math benchmarks with 7B and 14B models, but is not an inevitable byproduct of reasoning optimization: an SFT baseline preserves more than double the coverage, and staged SFT--DPO--RLVR pipelines retain early-step entropy. In summary, reasoning breadth is lost at the door, not inside the room. Code: this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.29188 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T07:28:26.726Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).