Metraipolis › archive › dc_ec4228ac852b
Learning Process Rewards via Reasoning State Propagation — arXiv:2609.39220
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.39220
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T12:42:35.419Z
- SHA-256 of the archived text
05d2348945069e1ea753f9846a728838fb5c6e924f4c53e8ff1bd1eaf5b9868a- Stable URL of this page
https://metraipolis.com/documento/dc_ec4228ac852b
How to cite
Learning Process Rewards via Reasoning State Propagation — arXiv:2609.39220 — arXiv (en), https://arxiv.org/abs/2609.39220 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ec4228ac852b (Metraipolis archive dc_ec4228ac852b, sha256 05d2348945069e1ea753f9846a728838fb5c6e924f4c53e8ff1bd1eaf5b9868a)
BibTeX
@misc{dcec4228ac852b,
title = {Learning Process Rewards via Reasoning State Propagation — arXiv:2609.39220},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.39220},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ec4228ac852b; sha256 05d2348945069e1ea753f9846a728838fb5c6e924f4c53e8ff1bd1eaf5b9868a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39220. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Process reward models (PRMs) have demonstrated notable effectiveness in test-time scaling and reinforcement learning by providing fine-grained signals for evaluating intermediate reasoning states, but their training relies heavily on costly process annotations. A natural way to alleviate this dependence is to complement limited process supervision with scalable outcome supervision. However, existing PRMs often model reasoning prefixes independently, providing no explicit mechanism for effectively using final outcome to guide the learning of intermediate reasoning states. We introduce Reasoning State Propagation (RSP), which represents each reasoning prefix with a binary validity state and models transitions between successive states across the reasoning trajectory. Specifically, RSP predicts a break probability that a valid state becomes invalid and a repair probability that an invalid state returns to valid. By propagating these transitions, RSP connects intermediate states to the final state, allowing process annotations to supervise intermediate states while outcome labels supervise the final state and can provide learning signals to preceding steps. Across reasoning search, response selection, and reinforcement learning, RSP consistently outperforms representative PRM baselines, with average improvements over Qwen2.5-Math-PRM of 5.6% in beam search and 2.1% in reinforcement learning. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.39220 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T12:14:13.743Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).