Metraipolis › archive › dc_3480eb68c478
GRPODropout: Less is More for Online Reinforcement Learning Rollouts — arXiv:2610.11854
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.11854
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T06:03:18.597Z
- SHA-256 of the archived text
e3041c43c4b5a4bf5a533bc4fdea1b1f282c034360902a64373c2a9f19e5230a- Stable URL of this page
https://metraipolis.com/documento/dc_3480eb68c478
How to cite
GRPODropout: Less is More for Online Reinforcement Learning Rollouts — arXiv:2610.11854 — arXiv (en), https://arxiv.org/abs/2610.11854 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3480eb68c478 (Metraipolis archive dc_3480eb68c478, sha256 e3041c43c4b5a4bf5a533bc4fdea1b1f282c034360902a64373c2a9f19e5230a)
BibTeX
@misc{dc3480eb68c478,
title = {GRPODropout: Less is More for Online Reinforcement Learning Rollouts — arXiv:2610.11854},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.11854},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3480eb68c478; sha256 e3041c43c4b5a4bf5a533bc4fdea1b1f282c034360902a64373c2a9f19e5230a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11854. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reinforcement learning (RL) methods such as GRPO substantially improve large language model reasoning but often suffer from policy entropy collapse: the loss of sampling diversity weakens exploration and limits further improvement. Existing methods address this issue either through algorithm-level interventions, such as reward modification and entropy/KL regularization, or through token-level reweighting. We investigate a complementary perspective: entropy collapse can also be mitigated by changing which generated rollouts contribute to policy updates. Under the same sampling budget, not all rollouts contribute positively to an update, and selectively excluding some can improve learning. To address this, we propose GRPODropout: before the standard update, we use a simple strategy that selectively removes a small number of high-probability positive-advantage rollouts and recenters the retained advantages. To motivate this design, we develop a rollout-level theoretical analysis that guides method design and threshold selection. The method changes only rollout usage, and adds negligible computational overhead. Experiments show higher accuracy than original GRPO and higher actor entropy while using fewer rollout samples for updates, illustrating "less is more." This work provides insight into RL rollout usage: removing some rollouts can improve performance. Code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.11854 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T05:47:01.907Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).