Metraipolis › archive › dc_2ebbaa70362c

Improved Sample Complexity Analysis For First Order Bilevel Reinforcement Learning — arXiv:2611.01500

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.01500
License (as archived)
CC0-1.0
Archived
2026-11-03T10:26:26.790Z
SHA-256 of the archived text
e057f49b2ee5ddad3e12cbb7b03d044d8c2c19c2484182416dc800fb6b4551c1
Stable URL of this page
https://metraipolis.com/documento/dc_2ebbaa70362c

How to cite

Improved Sample Complexity Analysis For First Order Bilevel Reinforcement Learning — arXiv:2611.01500 — arXiv (en), https://arxiv.org/abs/2611.01500 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2ebbaa70362c (Metraipolis archive dc_2ebbaa70362c, sha256 e057f49b2ee5ddad3e12cbb7b03d044d8c2c19c2484182416dc800fb6b4551c1)
BibTeX
@misc{dc2ebbaa70362c,
  title = {Improved Sample Complexity Analysis For First Order Bilevel Reinforcement Learning — arXiv:2611.01500},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01500},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2ebbaa70362c; sha256 e057f49b2ee5ddad3e12cbb7b03d044d8c2c19c2484182416dc800fb6b4551c1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01500. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Bilevel reinforcement learning optimizes an upper-level objective through the policy response to a lower-level reinforcement learning problem. Such problems arise when learning a reward model while adapting a policy to that reward. However, a gap remains between sample-complexity guarantees and practical actor--critic training. Tabular analyses restrict applicability to finite spaces, while continuous-state analyses often impose a uniform Polyak--Łojasiewicz (PL) condition. We show that this condition need not hold even for smooth positive policies with exactly representable values. We develop a single-loop, fully first-order method for continuous states and finite actions with specified multilayer neural actors and a persistently trained nonlinear critic built from fixed features. Under explicit initialization, regularity, feature-excitation, and value-realizability conditions, fixed reference-KL regularization yields local curvature, and we prove containment of the coupled iterates. Shared trajectories control sampling noise, while paired critic analysis controls normalized critic-error differences as the response perturbation shrinks. A joint descent and tracking argument establishes an improved $\widetilde{\mathcal O}(\varepsilon^{-2})$ sample complexity for $O(\varepsilon)$ expected squared gradient norm of the upper objective induced by the selected regularized policy response, including critic training.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.01500 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:19:50.084Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).