Metraipolis › archive › dc_ffcea21cffbd

Priority-Preserving Advantage Aggregation for Multi-Reward Policy Optimization — arXiv:2610.34862

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.34862
License (as archived): CC0-1.0
Archived: 2026-10-27T08:13:12.559Z
SHA-256 of the archived text: 8da0a6ebcb3a75561c466c3ea9c118c7f9305c53929b8b016ed42fe1639ec041
Stable URL of this page: https://metraipolis.com/documento/dc_ffcea21cffbd

How to cite

Priority-Preserving Advantage Aggregation for Multi-Reward Policy Optimization — arXiv:2610.34862 — arXiv (en), https://arxiv.org/abs/2610.34862 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ffcea21cffbd (Metraipolis archive dc_ffcea21cffbd, sha256 8da0a6ebcb3a75561c466c3ea9c118c7f9305c53929b8b016ed42fe1639ec041)

BibTeX

@misc{dcffcea21cffbd,
  title = {Priority-Preserving Advantage Aggregation for Multi-Reward Policy Optimization — arXiv:2610.34862},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.34862},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ffcea21cffbd; sha256 8da0a6ebcb3a75561c466c3ea9c118c7f9305c53929b8b016ed42fe1639ec041},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.34862. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM post-training often optimizes multiple objectives with unequal priorities, such as correctness and reasoning efficiency. However, the widely used additive reward aggregation in RL can violate these priorities: when primary and auxiliary advantages conflict, a sufficiently large auxiliary signal can overturn the preference induced by the primary objective. We term this problem \emph{advantage priority inversion}. Although reward-level gating can protect priority more directly, it still removes potentially useful auxiliary distinctions among rollout samples that fail the primary condition. To address this trade-off, we propose Primary Advantage Policy Optimization (\textbf{PAPO}), a priority-aware advantage aggregation method that asymmetrically combines a primary and an auxiliary advantage. PAPO retains auxiliary supervision when the objectives agree and smoothly attenuates conflicting auxiliary signals, preserving the primary sign during fusion for \emph{primary-consistent ordering}. This rule applies to sequence-level advantages in GRPO and token-level advantages in actor--critic PPO. Experiments on multiple domains and models show that PAPO consistently achieves the best primary objective while maintaining best or competitive auxiliary-objective performance across both GRPO- and PPO-based optimization. \href{ this https URL }{Code is available}.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.34862 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:56:00.312Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).