Metraipolis › archive › dc_dbc6e7df78ff
When Does Momentum Help in Reinforcement Learning with Verifiable Rewards? — arXiv:2610.20413
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.20413
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T14:20:22.947Z
- SHA-256 of the archived text
4f75079f246f56c0de3849eb274acc0652808e694b5b6ad973fce77bb4f31ca6- Stable URL of this page
https://metraipolis.com/documento/dc_dbc6e7df78ff
How to cite
When Does Momentum Help in Reinforcement Learning with Verifiable Rewards? — arXiv:2610.20413 — arXiv (en), https://arxiv.org/abs/2610.20413 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dbc6e7df78ff (Metraipolis archive dc_dbc6e7df78ff, sha256 4f75079f246f56c0de3849eb274acc0652808e694b5b6ad973fce77bb4f31ca6)
BibTeX
@misc{dcdbc6e7df78ff,
title = {When Does Momentum Help in Reinforcement Learning with Verifiable Rewards? — arXiv:2610.20413},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.20413},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dbc6e7df78ff; sha256 4f75079f246f56c0de3849eb274acc0652808e694b5b6ad973fce77bb4f31ca6},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20413. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Momentum is a classical tool for accelerating convex optimization and a standard component of modern large language model training. Yet recent empirical work has questioned the usefulness of first-moment momentum in reinforcement learning with verifiable rewards (RLVR). We study this question through a bandit formulation of RLVR with bias-corrected exponential moving average (EMA) momentum, where the EMA coefficient $\beta$ controls how quickly the weights on past gradients decay. When policy gradients are evaluated exactly, we prove that increasing $\beta$ within a range that prevents overshoot strictly accelerates optimization. When gradients are estimated from sampled responses, a positive expected gain can be obscured by sampling fluctuations. We show that as the learning rate $\eta$ decreases, the expected gain with fixed $\beta$ vanishes faster than these fluctuations, whereas increasing $\beta$ to keep $\eta/(1-\beta)$ within the safe regime preserves a nonvanishing gain that eventually dominates them. Bandit simulations quantitatively validate these predictions. RLVR experiments on Qwen3-4B-Base using SGD with EMA and AdamW support our principle of choosing larger coefficients in practical LLM post-training. On Qwen2.5-32B with AdamW, simply increasing $\beta_1$ improves mean validation accuracy by nearly $2$ percentage points over the conventional choice, demonstrating the principle's effectiveness at a larger model scale. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Machine Learning (stat.ML). Pagina del lavoro: https://arxiv.org/abs/2610.20413 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T13:53:35.746Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).