Metraipolis › archive › dc_df6afbe17caf

Hierarchical Reinforcement Learning with Stable Temporal Abstraction for Language Model Agents — arXiv:2610.05473

Source
arXiv
Original
https://arxiv.org/abs/2610.05473
License (as archived)
CC0-1.0
Archived
2026-10-06T05:25:11.957Z
SHA-256 of the archived text
834b3c0e40da3bf2f5e9729f4c83378c0d955dab82b2eb5e84fc1ba3bd0e70e1
Stable URL of this page
https://metraipolis.com/documento/dc_df6afbe17caf

How to cite

Hierarchical Reinforcement Learning with Stable Temporal Abstraction for Language Model Agents — arXiv:2610.05473 — arXiv, https://arxiv.org/abs/2610.05473 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_df6afbe17caf (Metraipolis archive dc_df6afbe17caf, sha256 834b3c0e40da3bf2f5e9729f4c83378c0d955dab82b2eb5e84fc1ba3bd0e70e1)
BibTeX
@misc{dcdf6afbe17caf,
  title = {Hierarchical Reinforcement Learning with Stable Temporal Abstraction for Language Model Agents — arXiv:2610.05473},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05473},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_df6afbe17caf; sha256 834b3c0e40da3bf2f5e9729f4c83378c0d955dab82b2eb5e84fc1ba3bd0e70e1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05473. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Hierarchical reinforcement learning improves long-horizon control by organizing primitive actions around persistent subgoals and assigning credit at multiple temporal scales. Recent hierarchical language agents bring these benefits to interactive tasks by explicitly separating subgoal planning from action execution. We observe, however, that an explicit hierarchy does not by itself determine how stable the resulting temporal abstraction is: the learned boundary policy may replace the subgoal almost every turn, making it effectively transient, or retain a subgoal after it has stopped being appropriate. We call this temporal abstraction instability. We propose Stable Temporal Abstraction via Constrained Optimization (STAC), a constrained boundary-policy optimization method that represents premature replanning and stale persistence as constraint costs. STAC applies the resulting Lagrangian costs only to the sampled boundary decision, leaving the underlying algorithm's rewards, critic targets, subgoal advantages, and primitive-action advantages unchanged. Across two backbones and two benchmarks, STAC improves success over a strong hierarchical baseline by $8.1$ and $7.9$ points on ALFWorld and WebShop with Qwen3-0.6B, and by $23.5$ and $15.8$ points with Llama-3.2-1B-Instruct.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.05473 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:05.593Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).