Metraipolis › archive › dc_837bc84d4f59

Efficient Hypergradient Descent for Inverse Reinforcement Learning — arXiv:2608.11052

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.11052
License (as archived)
CC0-1.0
Archived
2026-10-19T06:19:32.999Z
SHA-256 of the archived text
3d37e7ed73620def7926a62462314f459d7dc604a480f316f23ce132afc2f271
Stable URL of this page
https://metraipolis.com/documento/dc_837bc84d4f59

How to cite

Efficient Hypergradient Descent for Inverse Reinforcement Learning — arXiv:2608.11052 — arXiv (en), https://arxiv.org/abs/2608.11052 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_837bc84d4f59 (Metraipolis archive dc_837bc84d4f59, sha256 3d37e7ed73620def7926a62462314f459d7dc604a480f316f23ce132afc2f271)
BibTeX
@misc{dc837bc84d4f59,
  title = {Efficient Hypergradient Descent for Inverse Reinforcement Learning — arXiv:2608.11052},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.11052},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_837bc84d4f59; sha256 3d37e7ed73620def7926a62462314f459d7dc604a480f316f23ce132afc2f271},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.11052. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Inverse reinforcement learning (IRL) aims to recover a reward function under which the resulting policy reproduces the behavior observed in expert demonstrations. A natural approach is to formulate IRL as a bilevel optimization problem, in which the inner level corresponds to policy optimization under the learned reward and the outer level measures the discrepancy between the induced policy and expert data. However, this formulation is computationally challenging in practice because the outer update requires a hypergradient involving an inverse-Hessian-vector product for the inner objective. We address this challenge by showing that, at the inner optimum, the Hessian of the inner objective is proportional to the Fisher information matrix of the policy, yielding a structured Fisher-based hypergradient closely related to Natural Hypergradient Descent. To address the resulting scalability bottleneck associated with large Fisher matrices, we approximate the required inverse-Fisher-vector product using a streaming spectral sketch, avoiding explicit construction of the Fisher matrix. We evaluate our approach against a first-order stochastic bilevel baseline across discrete- and continuous-control environments. The results demonstrate competitive policy performance and strong reward-ranking quality, while Fisher sketching reduces curvature-storage complexity and can improve computational efficiency relative to an explicit Fisher solver.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-12 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2608.11052 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T06:11:45.106Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).