Metraipolis › archive › dc_d7b59a4aed2c

GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models — arXiv:2608.17411

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.17411
License (as archived)
CC0-1.0
Archived
2026-10-18T14:48:36.920Z
SHA-256 of the archived text
ecc52bc1f7f1f3a5fdece9ba08470f4cd046a84b9774e0d089cf4fdf262f8947
Stable URL of this page
https://metraipolis.com/documento/dc_d7b59a4aed2c

How to cite

GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models — arXiv:2608.17411 — arXiv (en), https://arxiv.org/abs/2608.17411 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d7b59a4aed2c (Metraipolis archive dc_d7b59a4aed2c, sha256 ecc52bc1f7f1f3a5fdece9ba08470f4cd046a84b9774e0d089cf4fdf262f8947)
BibTeX
@misc{dcd7b59a4aed2c,
  title = {GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models — arXiv:2608.17411},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.17411},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d7b59a4aed2c; sha256 ecc52bc1f7f1f3a5fdece9ba08470f4cd046a84b9774e0d089cf4fdf262f8947},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.17411. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Group Relative Policy Optimization (GRPO) has become a widely used approach for post-training Large Language Models (LLMs) for reasoning. In GRPO, the group gradients induced by different queries within the same mini-batch are directly averaged to form the policy update. However, these group gradients can point in conflicting directions. Our empirical analysis suggests that group-gradient conflicts tend to be associated with less effective policy updates, motivating the need for a reliable aggregated update direction under such conflicts. Standard GRPO aggregation treats the realized group gradients as deterministic contributions and does not account for differences in their reliability during aggregation. To address this issue, we propose Gradient Uncertainty-Aware Policy Optimization (GUPO), which models each group gradient as a random variable under a Bayesian formulation and estimates its probability distribution. GUPO then derives gradient uncertainty using a Dirichlet-based formulation and uses it to calibrate the contribution of each group gradient during aggregation. Extensive experiments on multiple benchmarks demonstrate the effectiveness of GUPO.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-19 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.17411 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T14:47:59.397Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-19?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).