Metraipolis › archive › dc_a3529611e609
EasyPPO: Stabilizing the Critic Is Key — arXiv:2609.36802
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.36802
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T17:29:30.492Z
- SHA-256 of the archived text
c8d6e4f93966f4474bf072363589277dfcda7bf2efd94dea06dd1429ac411a88- Stable URL of this page
https://metraipolis.com/documento/dc_a3529611e609
How to cite
EasyPPO: Stabilizing the Critic Is Key — arXiv:2609.36802 — arXiv (en), https://arxiv.org/abs/2609.36802 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a3529611e609 (Metraipolis archive dc_a3529611e609, sha256 c8d6e4f93966f4474bf072363589277dfcda7bf2efd94dea06dd1429ac411a88)
BibTeX
@misc{dca3529611e609,
title = {EasyPPO: Stabilizing the Critic Is Key — arXiv:2609.36802},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.36802},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a3529611e609; sha256 c8d6e4f93966f4474bf072363589277dfcda7bf2efd94dea06dd1429ac411a88},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36802. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
A key strength of Proximal Policy Optimization (PPO) is its learned critic, which uses historical trajectories collected during reinforcement learning to estimate expected returns and reduce policy-gradient variance. However, we find that the critic is also a major source of instability in reinforcement learning for large language models (LLMs). We identify two critic failure modes that destabilize PPO. First, filtering truncated rollouts from both actor and critic shifts the policy objective to reward conditioned on completion, allowing truncation to increase even as conditional reward improves. Second, heterogeneous return noise can cause high-variance prompts to dominate critic updates in finite batches. We introduce EasyPPO to address these failures. Actor-only overlong filtering trains the critic on returns from both completed and truncated rollouts. Noise-normalized critic regression weights each prompt's critic loss by the inverse standard deviation of its sampled returns, balancing noise contributions across prompts. Moderately smaller critic mini-batches confine outlier influence to fewer rollouts during gradient clipping. Across continuous-reward coding on FrontierCS, binary-reward mathematical reasoning on AIME24, and multi-turn search on Search-R1, EasyPPO remains stable throughout the full training horizon and consistently outperforms vanilla PPO, VAPO, and HL-Gauss PPO. Its best validation scores show relative gains of 14.89%, 2.28%, and 9.47% over PPO, respectively. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.36802 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T16:59:48.242Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).