Metraipolis › archive › dc_1207879d2ff4

SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learning — arXiv:2609.36552

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.36552
License (as archived)
CC0-1.0
Archived
2026-10-07T16:26:02.417Z
SHA-256 of the archived text
e40b33a2ef97ec2e31a9a9d1ceb26b4a4f3525d94e9faf67f7a589c97d63273d
Stable URL of this page
https://metraipolis.com/documento/dc_1207879d2ff4

How to cite

SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learning — arXiv:2609.36552 — arXiv (en), https://arxiv.org/abs/2609.36552 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1207879d2ff4 (Metraipolis archive dc_1207879d2ff4, sha256 e40b33a2ef97ec2e31a9a9d1ceb26b4a4f3525d94e9faf67f7a589c97d63273d)
BibTeX
@misc{dc1207879d2ff4,
  title = {SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learning — arXiv:2609.36552},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.36552},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1207879d2ff4; sha256 e40b33a2ef97ec2e31a9a9d1ceb26b4a4f3525d94e9faf67f7a589c97d63273d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36552. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Maximum Likelihood Reinforcement Learning (MaxRL) targets prompt-wise log-success and has shown strong performance on reasoning tasks. Under finite rollout budgets, however, the estimator used by MaxRL attenuates each prompt's likelihood gradient by a factor that depends on its success probability and rollout count. Under uniform rollout allocation, the common rollout count fails to compensate for success-dependent attenuation, leaving low-success prompts more strongly attenuated and distorting their relative contributions to the expected aggregate gradient. We introduce SERA (Scale-Equalized Rollout Allocation), which redistributes a fixed rollout budget to approximately equalize these finite-rollout scaling factors. Building on our theoretical analysis of how finite rollouts distort prompt-wise likelihood gradients, we formulate the allocation as a fixed-budget max--min problem, derive a waterline solution to its continuous relaxation, and introduce a multiplicity correction to remove the additional prompt weighting induced by heterogeneous rollout counts. Experiments show stronger alignment with exact likelihood gradients in a controlled ImageNet setting and improved multi-sample solution coverage over MaxRL on maze navigation and mathematical reasoning under matched training rollout budgets.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.36552 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:53.857Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).