Metraipolis › archive › dc_48bbcfbb7990

Li$^{2}$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping — arXiv:2610.40726

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.40726
License (as archived): CC0-1.0
Archived: 2026-11-03T08:04:50.111Z
SHA-256 of the archived text: 6a4991e35746b09991047cbe84ddd0957a0d31d5a04121a498fac7f7eb4073f9
Stable URL of this page: https://metraipolis.com/documento/dc_48bbcfbb7990

How to cite

Li$^{2}$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping — arXiv:2610.40726 — arXiv (en), https://arxiv.org/abs/2610.40726 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_48bbcfbb7990 (Metraipolis archive dc_48bbcfbb7990, sha256 6a4991e35746b09991047cbe84ddd0957a0d31d5a04121a498fac7f7eb4073f9)

BibTeX

@misc{dc48bbcfbb7990,
  title = {Li$^2$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping — arXiv:2610.40726},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.40726},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_48bbcfbb7990; sha256 6a4991e35746b09991047cbe84ddd0957a0d31d5a04121a498fac7f7eb4073f9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40726. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Asynchronous reinforcement learning (RL) is becoming a mainstream paradigm for post-training large language models (LLMs), as it improves training efficiency by updating the model as rollouts arrive. Recent work argues that group-wise sampling in GRPO does not naturally fit asynchronous RL training because rollout lengths follow a long-tailed distribution, and proposes Single-rollout Asynchronous Optimization (SAO) to improve stability and off-policy behavior, especially on agentic tasks. However, SAO relies on a closed-source value model obtained from scaling value pretraining. In this paper, we analyze GRPO and LiSAO, our lightweight reproduction of SAO without the pretrained value model, on single-turn reasoning and multi-turn agentic tasks. We find that GRPO outperforms LiSAO on reasoning tasks, while the reverse holds on agentic tasks. Since the value model of LiSAO is not yet well trained , LiSAO lags behind GRPO substantially in early training steps. To address this issue, we propose Li$^{2}$SAO, which Lifts Light Single-rollout Asynchronous Optimization with group-relative actor-critic bootstrapping: while the critic is cold, a group-relative advantage trains the actor and a group-centered return trains the critic, after which training switches to single-rollout GAE. Experiments show that Li$^{2}$SAO catches up with or surpasses GRPO in early training and further raises the later performance ceiling. Code is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.40726 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:59:53.574Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).