Metraipolis › archive › dc_783166ec662c

Provably Efficient Q-learning for Robust Reinforcement Learning — arXiv:2610.35656

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.35656
License (as archived)
CC0-1.0
Archived
2026-10-27T08:03:21.519Z
SHA-256 of the archived text
1a15a5dec23c2653c50456d989b2ff597fb982c9d3b8cb8fa849cb2f54e39cc4
Stable URL of this page
https://metraipolis.com/documento/dc_783166ec662c

How to cite

Provably Efficient Q-learning for Robust Reinforcement Learning — arXiv:2610.35656 — arXiv (en), https://arxiv.org/abs/2610.35656 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_783166ec662c (Metraipolis archive dc_783166ec662c, sha256 1a15a5dec23c2653c50456d989b2ff597fb982c9d3b8cb8fa849cb2f54e39cc4)
BibTeX
@misc{dc783166ec662c,
  title = {Provably Efficient Q-learning for Robust Reinforcement Learning — arXiv:2610.35656},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.35656},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_783166ec662c; sha256 1a15a5dec23c2653c50456d989b2ff597fb982c9d3b8cb8fa849cb2f54e39cc4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.35656. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Standard reinforcement learning (RL) algorithms based on Markov Decision Processes (MDPs) are vulnerable to sim-to-real gaps. To address this issue, distributionally robust RL algorithms based on robust MDPs (RMDPs) have been proposed, for ensuring that agents can perform reliably to unpredictable conditions in real-world environments. Despite of the rich literature, \revision{an efficient model-free method is still lacking} for robust RL, particularly for Markovian data, which is more relevant in practical applications. In this work, we take a step towards settling this problem by proposing a sample efficient (asynchronous Q-learning) algorithm for robust RL. In particular, consider a $\gamma$-discounted infinite-horizon RMDP with state space $\mathcal{S}$ and action space $\mathcal{A}$, where the uncertainty set is specified by the total variation (TV) distance with uncertainty level $\sigma$. Assuming access to a single trajectory of Markovian samples induced by a behavior policy and the nominal transition kernel, the sample complexity of the proposed Q-learning algorithm --- to achieve an entrywise $\varepsilon$-approximation of the optimal Q-function --- is bounded by $$ \frac{1}{\mu_{\mathsf{min}}\varepsilon^2(1-\gamma)^3\max\{1-\gamma,\sigma\}} + \frac{t_{\mathsf{mix}}}{\mu_{\mathsf{min}}(1-\gamma)} $$ up to logarithmic factors. Here, $t_{\mathsf{mix}}$ and $\mu_{\mathsf{min}}$ denote the mixing time and the minimum state-action occupancy probability of the sample trajectory, respectively. This result is nearly minimax optimal, with an additional factor of $\frac{1}{1-\gamma}$. Moreover, it achieves a significant improvement over existing bounds.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (stat.ML) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Statistics Theory (math.ST).
Pagina del lavoro: https://arxiv.org/abs/2610.35656 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:55:31.005Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).