Metraipolis › archive › dc_ead88049c5ae
Towards Better Training Signal: Advantage Clipped Policy Optimization — arXiv:2609.36816
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.36816
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T17:29:32.035Z
- SHA-256 of the archived text
51bd89708700cb1d209455a260b6748bc100d4d56c04aba21cee8d67245c1c64- Stable URL of this page
https://metraipolis.com/documento/dc_ead88049c5ae
How to cite
Towards Better Training Signal: Advantage Clipped Policy Optimization — arXiv:2609.36816 — arXiv (en), https://arxiv.org/abs/2609.36816 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ead88049c5ae (Metraipolis archive dc_ead88049c5ae, sha256 51bd89708700cb1d209455a260b6748bc100d4d56c04aba21cee8d67245c1c64)
BibTeX
@misc{dcead88049c5ae,
title = {Towards Better Training Signal: Advantage Clipped Policy Optimization — arXiv:2609.36816},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.36816},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ead88049c5ae; sha256 51bd89708700cb1d209455a260b6748bc100d4d56c04aba21cee8d67245c1c64},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36816. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reinforcement learning (RL) has become a cornerstone for improving the reasoning capabilities of large language models (LLMs), but the need for on-policy data substantially limits training efficiency. Reusing off-policy data through importance sampling (IS) can improve efficiency but introduce considerable instability. Hence, algorithms such as PPO and GRPO widely adopt IS-ratio clipping to stabilize training. However, training stability and gradient estimate are mainly determined by the product of IS ratio and advantage. To further stabilize training, we propose ACPO, which clips the product of the IS ratio and the advantage, leading to more stable gradient estimates. We also establish a connection between ACPO and gradient clipping in policy mirror descent (PMD), which is a standard technique to stabilize optimization process, and prove the convergence of clipped-PMD under the standard RL setting. Experiments on widely used mathematical reasoning benchmarks show that ACPO consistently outperforms PPO and GRPO in both accuracy and training efficiency, delivering 4-6 percentage points gains on standard math benchmarks, with Qwen3-8B+PPO. Hence, ACPO is a practical and effective alternative to conventional IS-ratio clipping for RL post-training of LLMs. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.36816 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T16:59:48.246Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).