Metraipolis › archive › dc_d9984190cb64
Bellman Policy Optimization — arXiv:2609.15987
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.15987
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T23:17:21.749Z
- SHA-256 of the archived text
ce26b2edbe6a246d0d697b82a9d7c059ddd2c7d917560e4936539586879ef258- Stable URL of this page
https://metraipolis.com/documento/dc_d9984190cb64
How to cite
Bellman Policy Optimization — arXiv:2609.15987 — arXiv (en), https://arxiv.org/abs/2609.15987 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d9984190cb64 (Metraipolis archive dc_d9984190cb64, sha256 ce26b2edbe6a246d0d697b82a9d7c059ddd2c7d917560e4936539586879ef258)
BibTeX
@misc{dcd9984190cb64,
title = {Bellman Policy Optimization — arXiv:2609.15987},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.15987},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d9984190cb64; sha256 ce26b2edbe6a246d0d697b82a9d7c059ddd2c7d917560e4936539586879ef258},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15987. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models (LLMs). We introduce Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective. The reformulation avoids estimating state values at intermediate states. We prove that it has the same unique optimal solution as the original PMD objective. We derive the practical BPO loss by approximating this objective. Its mismatch-correction weight is a smoothed ratio of complementary token probabilities. Experiments on mathematical reasoning benchmarks demonstrate the effectiveness of BPO. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.CL). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL); Optimization and Control (math.OC). Pagina del lavoro: https://arxiv.org/abs/2609.15987 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T22:51:10.043Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).