Metraipolis › archive › dc_aa9570f86655
CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning — arXiv:2609.36820
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.36820
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T16:57:36.274Z
- SHA-256 of the archived text
2edc3e71634f966affbfe7767f6677e40e459f3827d096354a64836f1205a7e9- Stable URL of this page
https://metraipolis.com/documento/dc_aa9570f86655
How to cite
CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning — arXiv:2609.36820 — arXiv (en), https://arxiv.org/abs/2609.36820 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aa9570f86655 (Metraipolis archive dc_aa9570f86655, sha256 2edc3e71634f966affbfe7767f6677e40e459f3827d096354a64836f1205a7e9)
BibTeX
@misc{dcaa9570f86655,
title = {CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning — arXiv:2609.36820},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.36820},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aa9570f86655; sha256 2edc3e71634f966affbfe7767f6677e40e459f3827d096354a64836f1205a7e9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36820. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Group Relative Policy Optimization (GRPO) is widely used to train reasoning language models, where it computes advantages by centering and normalizing rewards across rollouts of the same prompt. For multiple rewards, GRPO sums the reward components and normalizes the total reward by its within-group standard deviation. The corresponding variance equals the sum of all pairwise reward covariances. For a fixed centered reward, larger aggregate covariance produces smaller advantages, and vice versa, allowing update magnitudes to adapt to reward dependence. However, correlated rewards with large scales can dominate this normalization and suppress signals from smaller-scale rewards. We propose Correlation-Normalized GRPO (CorrGRPO), which normalizes pairwise covariances into Pearson correlation coefficients. CorrGRPO keeps the centered total reward unchanged while balancing the influence of differently scaled rewards on the correlation-based normalization. This allows advantage magnitudes to adapt to reward correlations without the normalization being dominated by large-scale reward components. We compare CorrGRPO with GRPO and other variants on code generation, tool calling, and agent security, using models ranging from 0.5B to 8B parameters. These tasks all involve multiple rewards that can improve together or present tradeoffs. Results show improvements across three domains, including code generation, tool calling, and agent security. Our code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.CL). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.36820 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T16:27:03.434Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).