Metraipolis › archive › dc_a649f4bfbca2

$O(1/n)$ Convergence Rate for RVI $Q$-Learning — arXiv:2611.07676

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.07676
License (as archived): CC0-1.0
Archived: 2026-11-06T14:50:39.348Z
SHA-256 of the archived text: 8cf9fb460317d688e3fda0c16d00631a3bf7cf9c56f6554f0ae3d0a7fae6e45e
Stable URL of this page: https://metraipolis.com/documento/dc_a649f4bfbca2

How to cite

$O(1/n)$ Convergence Rate for RVI $Q$-Learning — arXiv:2611.07676 — arXiv (en), https://arxiv.org/abs/2611.07676 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a649f4bfbca2 (Metraipolis archive dc_a649f4bfbca2, sha256 8cf9fb460317d688e3fda0c16d00631a3bf7cf9c56f6554f0ae3d0a7fae6e45e)

BibTeX

@misc{dca649f4bfbca2,
  title = {$O(1/n)$ Convergence Rate for RVI $Q$-Learning — arXiv:2611.07676},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.07676},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a649f4bfbca2; sha256 8cf9fb460317d688e3fda0c16d00631a3bf7cf9c56f6554f0ae3d0a7fae6e45e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07676. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We establish finite-time convergence guarantee for synchronous relative value iteration (RVI) $Q$-learning for finite weakly communicating Markov decision processes (MDPs). For stochastic Krasnosel'ski\uı--Mann iterations driven by a non-expansive, linearly regular map on a finite-dimensional normed space, we establish last-iterate mean-squared residual bounds under martingale-difference noise whose conditional second moment grows at most linearly with the squared residual. Suitable polynomial stepsizes of order $n^{-a}$ yield an $O(n^{-a})$ rate for $0<a\leq1$. We verify linear regularity and the noise condition for the centered Bellman operator for finite weakly communicating MDPs. Applying the general result gives an $O(n^{-1})$ bound on the mean-squared Bellman span residual of RVI $Q$-learning iterates when a suitable stepsize sequence is used. The positive gap between optimal and suboptimal stationary deterministic policies then yields an $O(n^{-1})$ bound on the expected suboptimality of the greedy policy. Consequently, $O(\epsilon^{-1})$ synchronous iterations suffice for an $\epsilon$-optimal policy in expectation. These guarantees are instance-dependent, with constants that depend on the hitting-time parameters and the suboptimality gap of the second-best policy.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.07676 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:21:42.631Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).