Metraipolis › archive › dc_c19b938f3580

Looped Actor: Depth-Recurrent Reasoning Models for Reinforcement Learning — arXiv:2609.37432

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.37432
License (as archived): CC0-1.0
Archived: 2026-10-07T17:31:18.521Z
SHA-256 of the archived text: 2aa87d1026a468dd152ccd6e6a96594d740a3706dcb934594e6a60b5c42b0a6c
Stable URL of this page: https://metraipolis.com/documento/dc_c19b938f3580

How to cite

Looped Actor: Depth-Recurrent Reasoning Models for Reinforcement Learning — arXiv:2609.37432 — arXiv (en), https://arxiv.org/abs/2609.37432 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c19b938f3580 (Metraipolis archive dc_c19b938f3580, sha256 2aa87d1026a468dd152ccd6e6a96594d740a3706dcb934594e6a60b5c42b0a6c)

BibTeX

@misc{dcc19b938f3580,
  title = {Looped Actor: Depth-Recurrent Reasoning Models for Reinforcement Learning — arXiv:2609.37432},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.37432},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c19b938f3580; sha256 2aa87d1026a468dd152ccd6e6a96594d740a3706dcb934594e6a60b5c42b0a6c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37432. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Looped reasoning models repeatedly apply a shared set of parameters, enabling more computation without increasing the model size. These models also support input-dependent computation by dynamically deciding when to stop looping. Motivated by the recent success of looped transformers in language modeling and reasoning, we investigate whether dynamic looping can similarly benefit sequential decision-making. We provide a complexity-theoretic motivation for this approach by showing that there exist Markov decision processes in which a state-adaptive policy achieves the optimal return with asymptotically less expected computation than any optimal fixed-runtime policy. To learn compute-adaptive policies in practice, we introduce Looped Actor, a transformer-based policy that repeatedly refines a latent representation toward a fixed point using a shared computational block. This allows the model to allocate computation adaptively by varying the number of loops based on the current state. We evaluate Looped Actor on 22 tasks across six environments, ranging from combinatorial puzzles to robotic manipulation and spanning online and offline reinforcement learning (RL) with discrete and continuous actions. Looped Actor matches or exceeds the performance of an untied baseline with 16$\times$ more parameters, with the largest gains in environments where action selection requires substantial multistep planning. For the Boxoban environment, we find that the computation allocation is structured: the number of loops increases with the number of remaining pushes and future optimal pushes become increasingly predictable from the latent state over successive loops. Together, these results highlight actor looping as a simple and efficient way to equip RL agents with adaptive computation and improve their planning capabilities. Code is available at this https URL

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.37432 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T16:59:48.329Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).