Metraipolis › archive › dc_44ae7bb286e6
Robust General Utility for Reinforcement Learning — arXiv:2608.03562
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.03562
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T12:42:26.499Z
- SHA-256 of the archived text
93ec5d98c752c65f08e7479eebd6f7420a4a4186d09c5087fb06aa4fc8086f33- Stable URL of this page
https://metraipolis.com/documento/dc_44ae7bb286e6
How to cite
Robust General Utility for Reinforcement Learning — arXiv:2608.03562 — arXiv (en), https://arxiv.org/abs/2608.03562 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_44ae7bb286e6 (Metraipolis archive dc_44ae7bb286e6, sha256 93ec5d98c752c65f08e7479eebd6f7420a4a4186d09c5087fb06aa4fc8086f33)
BibTeX
@misc{dc44ae7bb286e6,
title = {Robust General Utility for Reinforcement Learning — arXiv:2608.03562},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.03562},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_44ae7bb286e6; sha256 93ec5d98c752c65f08e7479eebd6f7420a4a4186d09c5087fb06aa4fc8086f33},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03562. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reinforcement learning (RL) with general utility extends classic RL by optimizing an arbitrary utility functional of the policy-induced occupancy measure, thereby enabling a broader range of applications. However, previous work on general utility RL typically assumes the evaluation utility is fixed and correctly specified. In practice, the utility used at deployment can deviate from the training one, creating a robustness gap that prior work does not address. Motivated by this, we propose robust general-utility RL, a minimax learning framework that trains policies against utility misspecification within a prescribed uncertainty set. Our framework strictly generalizes standard general-utility RL while also providing a unified view of many existing RL frameworks, including reward-robust RL and constrained RL, through appropriate choices of the utility uncertainty set. We further develop provably convergent stochastic algorithms for two regimes. For concave utilities, we develop a projected stochastic gradient descent-ascent method and establish stationarity guarantees. For the more challenging nonconcave regime, we propose a stochastic prox-extragradient algorithm that mitigates ill-posed behavior induced by nonconcavity, with convergence guarantees to approximate first-order stationarity. Experiments on LLM safety alignment and exploration maximization tasks further corroborate the convergence behavior consistent with our theory. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.03562 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T12:31:30.358Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).