Metraipolis › archive › dc_37c8595255f1
Reward Budgeting Reduces Premature Convergence in Reinforcement Learning for LLM Reasoning — arXiv:2611.05942
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.05942
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-05T18:22:05.342Z
- SHA-256 of the archived text
7d80a5e61f530567186895d573c70f3cd1c21808da669040265dbaf47a22a056- Stable URL of this page
https://metraipolis.com/documento/dc_37c8595255f1
How to cite
Reward Budgeting Reduces Premature Convergence in Reinforcement Learning for LLM Reasoning — arXiv:2611.05942 — arXiv (en), https://arxiv.org/abs/2611.05942 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_37c8595255f1 (Metraipolis archive dc_37c8595255f1, sha256 7d80a5e61f530567186895d573c70f3cd1c21808da669040265dbaf47a22a056)
BibTeX
@misc{dc37c8595255f1,
title = {Reward Budgeting Reduces Premature Convergence in Reinforcement Learning for LLM Reasoning — arXiv:2611.05942},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.05942},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_37c8595255f1; sha256 7d80a5e61f530567186895d573c70f3cd1c21808da669040265dbaf47a22a056},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05942. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
RL for LLM reasoning often reaches an early performance plateau as the policy distribution becomes increasingly sharp. Recent work has studied policy entropy as both a diagnostic and a control signal for this behavior. In our experiments, however, increasing entropy alone does not reliably change the eventual plateau. We analyze RL through its implicit target policy and derive a sharpness factor $\kappa$, together with an effective counterpart $\kappa_{\mathrm{eff}}$, that describes how strongly reward or advantage gaps are translated into probability-ratio gaps. Motivated by this view, we introduce \emph{reward budgeting}, a plug-in advantage construction that assigns a fixed total reward budget to each prompt. For binary rewards and independent on-policy rollouts, its unclipped score-function estimator follows the gradient of an exact finite-$G$ concave objective with diminishing returns, while its advantage gap decreases as more rollouts become correct. For general rewards, a shifted log-mean interpretation holds asymptotically under the stated regularity conditions. These results motivate, rather than establish exact objective equivalence for, the clipped implementations. On Qwen3-8B and Llama3.2-3B-Instruct, reward budgeting consistently improves a range of RL baselines, with gains up to +12.50 on AMC23 and +10.00 on AIME24 and AIME25. Overall, our results support an update-dynamics view of premature convergence and show that strong gains can be obtained without explicitly preserving entropy.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.05942 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:15:49.008Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).