Metraipolis › archive › dc_c059cff6d8a6
Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO — arXiv:2609.36932
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.36932
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T16:23:16.841Z
- SHA-256 of the archived text
27c6ad45967579748c541b56a5ef612070f43e6c218891ca8e56c4978b2bb740- Stable URL of this page
https://metraipolis.com/documento/dc_c059cff6d8a6
How to cite
Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO — arXiv:2609.36932 — arXiv (en), https://arxiv.org/abs/2609.36932 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c059cff6d8a6 (Metraipolis archive dc_c059cff6d8a6, sha256 27c6ad45967579748c541b56a5ef612070f43e6c218891ca8e56c4978b2bb740)
BibTeX
@misc{dcc059cff6d8a6,
title = {Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO — arXiv:2609.36932},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.36932},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c059cff6d8a6; sha256 27c6ad45967579748c541b56a5ef612070f43e6c218891ca8e56c4978b2bb740},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36932. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Recently, Group Relative Policy Optimization (GRPO) and its variants have been developed for policy optimization and demonstrated notable performance gains. However, these methods usually incur substantial computational overhead due to per-question multi-rollout sampling and repeated per-token probability evaluation across rollouts. Furthermore, low-information or highly homogeneous trajectories can degrade downstream learning signal efficiency, hindering model optimization and limiting final performance. To address these issues, we propose FastRL, a novel plug-and-play reinforcement learning framework that simultaneously improves training efficiency and the effectiveness of policy learning. Specifically, 1) We introduce an advantage-aware pruning strategy to selectively preserve high-advantage trajectories while maximizing inter-trajectory gradient diversity. 2) Then, we design an adaptive rollout sampling mechanism to dynamically adjust the sampling scale across different training stages based on historical pruning distributions, balancing exploration adequacy and computational efficiency. Experiments demonstrate that FastRL can be seamlessly integrated into GRPO, DAPO, and GSPO variants, achieving an average 2.07$\times$ training speedup on Geometry3K and GeoQA8K-R1V, along with an approximately 1.64\% improvement in average accuracy on visual reasoning benchmarks. Source codes will be available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.36932 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T15:54:53.566Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).