Metraipolis › archive › dc_304389fe148a
On Semi-Markov Suboptimality in Hierarchical Reinforcement Learning — arXiv:2610.05338
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.05338
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T05:26:49.840Z
- SHA-256 of the archived text
d54413ae295cc751a363041350085dd52aea79bc55402fdd04f3ed7f81171f3f- Stable URL of this page
https://metraipolis.com/documento/dc_304389fe148a
How to cite
On Semi-Markov Suboptimality in Hierarchical Reinforcement Learning — arXiv:2610.05338 — arXiv, https://arxiv.org/abs/2610.05338 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_304389fe148a (Metraipolis archive dc_304389fe148a, sha256 d54413ae295cc751a363041350085dd52aea79bc55402fdd04f3ed7f81171f3f)
BibTeX
@misc{dc304389fe148a,
title = {On Semi-Markov Suboptimality in Hierarchical Reinforcement Learning — arXiv:2610.05338},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.05338},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_304389fe148a; sha256 d54413ae295cc751a363041350085dd52aea79bc55402fdd04f3ed7f81171f3f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05338. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Hierarchical reinforcement learning uses temporally extended subtasks for exploration, yet committing to their execution can restrict both deployment and policy learning. We identify and separate the resulting execution and policy suboptimality. Task and execution trees distinguish reward objectives from policy choices and decision interruption. A Unified Value Function for HRL and a four-stage Generalized Hierarchical Bellman Equation then support a common analysis of both losses. Under bounded rewards and uniform termination, we establish hierarchical policy and execution improvement results. With the remaining node policies fixed, task-subtree compatibility and node-policy optimality under the original execution mode establish when Markov execution is optimal. The resulting decomposition leads to independent execution choices for behavior, targets, and deployment. We instantiate this principle through execution improvement and one-stage or two-stage policy improvement at arbitrary hierarchy depth. Option-based and goal-conditioned experiments demonstrate complementary gains from changing execution and changing the learning target. Controlled stochastic environments show how these gains depend on stochastic transition strength and spatial structure. This framework makes execution design an explicit component of hierarchical policy optimization. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.05338 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T05:24:06.001Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).