Metraipolis › archive › dc_dd86faf68aba
Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guide — arXiv:2608.30051
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.30051
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T07:44:30.855Z
- SHA-256 of the archived text
69e5666cbcbb1cb29e5883a8a8a5e6e7082eb51a9e29b0d5273ce132b6bf705d- Stable URL of this page
https://metraipolis.com/documento/dc_dd86faf68aba
How to cite
Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guide — arXiv:2608.30051 — arXiv (en), https://arxiv.org/abs/2608.30051 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dd86faf68aba (Metraipolis archive dc_dd86faf68aba, sha256 69e5666cbcbb1cb29e5883a8a8a5e6e7082eb51a9e29b0d5273ce132b6bf705d)
BibTeX
@misc{dcdd86faf68aba,
title = {Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guide — arXiv:2608.30051},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.30051},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dd86faf68aba; sha256 69e5666cbcbb1cb29e5883a8a8a5e6e7082eb51a9e29b0d5273ce132b6bf705d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.30051. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Process reward models (PRMs) provide dense step-level guidance for search-based reasoning, enabling inference-time compute to be allocated toward promising partial solutions. However, recent evidence suggests that PRM-guided search can over-optimize imperfect process rewards, pruning viable trajectories while expanding spurious ones. In this work, we theoretically show that directly leveraging PRM score is vulnerable to verifier noise through an extreme-value effect: non-viable prefixes become more likely to receive spuriously high scores as reasoning depth increase. Therefore, we formulate the PRM-guided search as a robust optimization problem over plausible reward perturbations, termed maximin PRM-guided search, leading to a training-free robust process supervision method that preserves promising alternatives when step-level scores are noisy. Maximin PRM-guided search mitigates this failure mode by reducing sensitivity to over-optimized PRM outliers. Without fine-tuning or online adaptation, maximin search consistently improves the PRM-guided search by 17-35\% on average, outperforming outcome- and step-level baselines in 14 out of 16 settings. Our source code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.30051 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T07:28:26.582Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).