Metraipolis › archive › dc_ec7871176dd1
BoT-GRPO: Efficient Process-Reward RL for Reasoning via Bag-of-Token Aggregation — arXiv:2610.09804
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.09804
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T11:59:21.943Z
- SHA-256 of the archived text
c97b80ed33541df5f5e635420ecea86eeb78b067b9304bc5775c7dbe273526d9- Stable URL of this page
https://metraipolis.com/documento/dc_ec7871176dd1
How to cite
BoT-GRPO: Efficient Process-Reward RL for Reasoning via Bag-of-Token Aggregation — arXiv:2610.09804 — arXiv (en), https://arxiv.org/abs/2610.09804 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ec7871176dd1 (Metraipolis archive dc_ec7871176dd1, sha256 c97b80ed33541df5f5e635420ecea86eeb78b067b9304bc5775c7dbe273526d9)
BibTeX
@misc{dcec7871176dd1,
title = {BoT-GRPO: Efficient Process-Reward RL for Reasoning via Bag-of-Token Aggregation — arXiv:2610.09804},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.09804},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ec7871176dd1; sha256 c97b80ed33541df5f5e635420ecea86eeb78b067b9304bc5775c7dbe273526d9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09804. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reinforcement learning is now central to eliciting reasoning in large language models, while in the popular algorithm Group Relative Policy Optimization (GRPO) every token in a rollout receives the same advantage. We ask how to make process supervision efficient: accelerating convergence and improving final quality without the cost of value networks. We propose Bag-of-Tokens Group Relative Policy Optimization (BoT-GRPO), which extends GRPO to token-level reward models through a length-invariant "bag of tokens" aggregation: it collects all token-level rewards across rollouts, weights each by the inverse of its source sequence length, and computes per-token advantages relative to weighted group statistics. BoT-GRPO is critic-free, and is a drop-in replacement wherever GRPO is used when token-level reward is available. On React front-end code generation, BoT-GRPO reaches $80\%$ compile rate up to $1.9\times$ faster than GRPO and converges faster than modern GRPO variants (GSPO, DAPO, PURE) while reaching higher final compile and VLM-judged win rates. On a second task, AIME mathematical reasoning, BoT-GRPO delivers absolute Pass@$k$ gains up to $8.1\%$ over GRPO in half the steps. For both tasks we compare the algorithm's performance on reasoning vs. non-reasoning base-model families (Qwen2.5-3B, SmolLM3-3B, Phi-4-mini-reasoning). Our experiments also yield a practical recipe for the reward model itself: reward stability matters more than richness: clean, bounded, stable fine-grained signals consistently accelerate learning where noisier alternatives stall. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.09804 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T10:50:37.718Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).