Metraipolis › archive › dc_27474659e67b

BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL — arXiv:2609.09783

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.09783
License (as archived)
CC0-1.0
Archived
2026-10-16T07:26:18.350Z
SHA-256 of the archived text
dcfce9829e70ff64aec1c6d8ed2a5b0d49d29158307d026070512837eee9d736
Stable URL of this page
https://metraipolis.com/documento/dc_27474659e67b

How to cite

BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL — arXiv:2609.09783 — arXiv (en), https://arxiv.org/abs/2609.09783 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_27474659e67b (Metraipolis archive dc_27474659e67b, sha256 dcfce9829e70ff64aec1c6d8ed2a5b0d49d29158307d026070512837eee9d736)
BibTeX
@misc{dc27474659e67b,
  title = {BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL — arXiv:2609.09783},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.09783},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_27474659e67b; sha256 dcfce9829e70ff64aec1c6d8ed2a5b0d49d29158307d026070512837eee9d736},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.09783. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Asynchronous reinforcement learning has become the standard way to scale training for large language models (LLM), but the resulting policy lag biases the critic toward the stale behavior policy. Existing work on asynchronous LLM training corrects the actor and leaves this bias unaddressed, while the off-policy value correction of classical RL does not carry over to long-horizon agentic tasks, since a short correction horizon leaves the regression target free of the reward and a long one lets the product of importance ratios drift exponentially with the trajectory length. We propose BRACE, an anchored Bellman-residual correction for stale value models. BRACE bounds the correction horizon to a prefix of policy tokens and anchors a constant-weight Monte-Carlo tail beyond it, which separates policy correction from reward propagation. BRACE delivers a $9.8\%$ relative improvement in mean@1 on BrowseComp-Plus over the strongest baseline, runs $2.46\times$ faster per step than synchronous training, and remains stable $50$ updates off-policy.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-10 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.09783 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T07:00:16.347Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).