Metraipolis › archive › dc_5289065294c2
When Rewards Depend on the Group, GRPO Plays a Tournament — arXiv:2611.02404
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02404
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:11:51.992Z
- SHA-256 of the archived text
3a80eaf02659ebcec41b1323e30b85097d25e1aeb55c63464b5e5dcaeaf3dfbf- Stable URL of this page
https://metraipolis.com/documento/dc_5289065294c2
How to cite
When Rewards Depend on the Group, GRPO Plays a Tournament — arXiv:2611.02404 — arXiv (en), https://arxiv.org/abs/2611.02404 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5289065294c2 (Metraipolis archive dc_5289065294c2, sha256 3a80eaf02659ebcec41b1323e30b85097d25e1aeb55c63464b5e5dcaeaf3dfbf)
BibTeX
@misc{dc5289065294c2,
title = {When Rewards Depend on the Group, GRPO Plays a Tournament — arXiv:2611.02404},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02404},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5289065294c2; sha256 3a80eaf02659ebcec41b1323e30b85097d25e1aeb55c63464b5e5dcaeaf3dfbf},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02404. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Rewards for diversity, coverage and distribution matching make the reward of each sampled response depend on the rest of its group, and they are now routinely optimized with GRPO and RLOO. We prove that in this setting the group-relative baseline changes the objective. The expected update equals each response's own reward gradient plus $\alpha$ times its effect on its groupmates' rewards, with $\alpha=-1/(K-1)$ for groups of size $K$, so GRPO trains every response to beat its groupmates in a zero-sum tournament. It cannot see a pairwise diversity bonus when K=2, sees less than half of it at any K, and often does worse than using no baseline at all. Setting $\alpha=+1$ gives Team-GRPO, an unbiased estimator for any group-dependent reward that works with two samples per prompt and requires no extra model calls for pairwise rewards. The theory predicts training quantitatively. It locates the fixed points of a 9B policy to within 0.02 in $\ell_1$ and predicts how concentrated trained policies become with Pearson r=0.995. On GlobalOpinionQA, GRPO collapses onto the majority answer and ends further from human opinion than the untrained model, while Team-GRPO cuts the error from $0.20$ to $0.08$. On NoveltyBench, Team-GRPO avoids the diversity collapse of GRPO and significantly improves set utility. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2611.02404 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T10:59:52.414Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).