Metraipolis › archive › dc_3409283b04fa

GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation — arXiv:2609.21432

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.21432
License (as archived)
CC0-1.0
Archived
2026-10-10T01:05:17.937Z
SHA-256 of the archived text
bbd1818e295f13668b32c1dff28816fd08662948d8ba92d3c22e94cf39bdf0ce
Stable URL of this page
https://metraipolis.com/documento/dc_3409283b04fa

How to cite

GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation — arXiv:2609.21432 — arXiv (en), https://arxiv.org/abs/2609.21432 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3409283b04fa (Metraipolis archive dc_3409283b04fa, sha256 bbd1818e295f13668b32c1dff28816fd08662948d8ba92d3c22e94cf39bdf0ce)
BibTeX
@misc{dc3409283b04fa,
  title = {GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation — arXiv:2609.21432},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.21432},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3409283b04fa; sha256 bbd1818e295f13668b32c1dff28816fd08662948d8ba92d3c22e94cf39bdf0ce},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21432. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Post-training plays a pivotal role in enhancing the reasoning capabilities and task-specific expertise of large language models (LLMs). Despite recent advances in post-training methods, such as Group Relative Policy Optimization (GRPO), their practical deployment remains impeded by training instability arising from the reliance on importance sampling. We introduce Group Variance Policy Optimization (GVPO), a novel post-training method that integrates the analytical solution of KL-constrained reward maximization into its gradient weighting scheme. This formulation provides an intuitive interpretation: GVPO's gradient corresponds to the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) it guarantees a unique optimal solution, exactly to the KL-constrained reward maximization objective, and (2) it enables flexible sampling distributions without requiring importance sampling. Beyond general post-training, we show that GVPO naturally extends to on-policy distillation (OPD). Furthermore, GVPO enables the optimization of a broad family of extended OPD objectives, providing a principled foundation for diverse objective design. By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training and on-policy distillation.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.21432 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T00:23:42.452Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).