Metraipolis › archive › dc_96124d10879d
Benign Nonconvex Landscape for Policy Optimization: Infinite-Horizon Discounted MDPs with General State and Action Spaces — arXiv:2609.21433
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.21433
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-10T02:04:41.987Z
- SHA-256 of the archived text
83788a27d73278c2cdd2734691f06211958af628a144f425c2b3858f4684e69c- Stable URL of this page
https://metraipolis.com/documento/dc_96124d10879d
How to cite
Benign Nonconvex Landscape for Policy Optimization: Infinite-Horizon Discounted MDPs with General State and Action Spaces — arXiv:2609.21433 — arXiv (en), https://arxiv.org/abs/2609.21433 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_96124d10879d (Metraipolis archive dc_96124d10879d, sha256 83788a27d73278c2cdd2734691f06211958af628a144f425c2b3858f4684e69c)
BibTeX
@misc{dc96124d10879d,
title = {Benign Nonconvex Landscape for Policy Optimization: Infinite-Horizon Discounted MDPs with General State and Action Spaces — arXiv:2609.21433},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.21433},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_96124d10879d; sha256 83788a27d73278c2cdd2734691f06211958af628a144f425c2b3858f4684e69c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21433. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We study the optimization landscape for infinite-horizon discounted Markov decision processes (MDPs) with general state and action spaces under structured stationary policy classes. A general weighted policy-iteration approach to establishing global convergence guarantees for policy gradient methods requires closure under weighted policy improvement at every policy, a property that may fail even when the policy class contains an optimal policy. To address this issue, we propose weaker conditions that guarantee the absence of suboptimal stationary points and establish the Polyak--Lojasiewicz--Kurdyka (PLK) condition for the policy gradient objective with a finite concentrability coefficient. We also establish the PLK condition from a policy-improvement bound that holds at every state, without a concentrability assumption. Our general results encompass settings covered by the earlier framework when the common standing assumptions hold for the same policy class and parameter domain. We further verify our proposed conditions for two operations models: inventory systems with Markov-modulated demand and stochastic cash-balance problems. For both models, the Bellman equation yields approximate convexity of the Q-value functions in the action variable, with deviations controlled by the first-order stationarity measure. These estimates establish exponent-one and, under additional curvature assumptions, exponent-two PLK conditions, which, together with Lipschitz continuity of the policy gradient, imply an $\mathcal{O}(1/\epsilon)$ iteration complexity and linear convergence, respectively, for projected gradient descent using exact policy gradients. To the best of our knowledge, we provide the first non-asymptotic convergence rates for solving infinite-horizon discounted inventory systems with Markov-modulated demand and stochastic cash-balance problems using policy gradient methods.
Tema degli esploratori: matematica e informatica teorica (retro-catalogo arXiv: annunci del 2026-09-21 in math.OC).
Categorie arXiv: Optimization and Control (math.OC).
Pagina del lavoro: https://arxiv.org/abs/2609.21433 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T01:45:31.233Z dalla pagina https://arxiv.org/catchup/math.OC/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).