Metraipolis › archive › dc_f01abc1b03d9
Unregularized Convergence of Single-Loop, Entropy-Regularized Natural Actor-Critic — arXiv:2608.19587
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.19587
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T11:48:13.680Z
- SHA-256 of the archived text
39e2652082bd832308bdfb2132b6f5498643ee8a185c0812e9df4c4bc3954f00- Stable URL of this page
https://metraipolis.com/documento/dc_f01abc1b03d9
How to cite
Unregularized Convergence of Single-Loop, Entropy-Regularized Natural Actor-Critic — arXiv:2608.19587 — arXiv (en), https://arxiv.org/abs/2608.19587 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f01abc1b03d9 (Metraipolis archive dc_f01abc1b03d9, sha256 39e2652082bd832308bdfb2132b6f5498643ee8a185c0812e9df4c4bc3954f00)
BibTeX
@misc{dcf01abc1b03d9,
title = {Unregularized Convergence of Single-Loop, Entropy-Regularized Natural Actor-Critic — arXiv:2608.19587},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.19587},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f01abc1b03d9; sha256 39e2652082bd832308bdfb2132b6f5498643ee8a185c0812e9df4c4bc3954f00},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.19587. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
While entropy regularization is widely used to stabilize and accelerate Natural Policy Gradient methods, its ability to yield faster convergence rates for the unregularized objective remains underexplored. Existing analyses often rely on double-loop architectures and invoke a linear entropy penalty. To bridge the gap between theory and practice, we analyze a single-loop, entropy-regularized Natural Actor-Critic algorithm under compatible linear function approximation. By training an uncentered critic, our critic tracking can remain stable even as the training policy approaches determinism and the Fisher information matrix degenerates. We focus on two primary regimes for the optimization landscape: a Stochastic Regime, where we fuse coupled actor-critic updates into a joint Lyapunov recurrence, and a Deterministic Regime, where we pivot to a Policy Mirror Descent framework to circumvent the collapse of Euclidean geometry. By exploiting a positive Minimal Action Gap in the unregularized Markov decision process, we introduce an Exponential Translation mechanism that maps the regularized gap to the unregularized one up to an exponentially decaying tail. By tuning the fixed temperature, our algorithm achieves accelerated unregularized convergence rates, up to approximation-error terms: $\tilde{\mathcal{O}}(T_{total}^{-1})$ in the Stochastic Regime, and $\tilde{\mathcal{O}}(T_{total}^{-2/3})$ for the average iterate alongside $\tilde{\mathcal{O}}(T_{total}^{-1/3})$ for the last iterate in the Deterministic Regime. Here, $T_{total}$ denotes the total number of stochastic critic updates (or Monte Carlo rollouts). Furthermore, in the tabular setting, our positive-action-gap analysis yields a $\tilde{\mathcal{O}}(T_{total}^{-2/3})$ average-iterate rate, surpassing the $\mathcal{O}(T_{total}^{-1/2})$ worst-case statistical barrier that applies without a positive action margin.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-21 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.19587 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T11:47:43.306Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).