Metraipolis › archive › dc_f3ecd08168f4
Trust the Critic More — arXiv:2609.39247
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.39247
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T12:45:06.168Z
- SHA-256 of the archived text
e847c6f3048e92aa171dc3bb0bb2553be3f11b08a808595fed1ecf74e3399265- Stable URL of this page
https://metraipolis.com/documento/dc_f3ecd08168f4
How to cite
Trust the Critic More — arXiv:2609.39247 — arXiv (en), https://arxiv.org/abs/2609.39247 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f3ecd08168f4 (Metraipolis archive dc_f3ecd08168f4, sha256 e847c6f3048e92aa171dc3bb0bb2553be3f11b08a808595fed1ecf74e3399265)
BibTeX
@misc{dcf3ecd08168f4,
title = {Trust the Critic More — arXiv:2609.39247},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.39247},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f3ecd08168f4; sha256 e847c6f3048e92aa171dc3bb0bb2553be3f11b08a808595fed1ecf74e3399265},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39247. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Standard language model RL algorithms credit every token of a long rollout with the same advantage determined by the terminal reward. Actor-critic methods can provide finer-grained credit assignment, but learned critics are generally considered too inaccurate to trust when training LLMs with RL. In recent works, even when a critic is present, it is used only for baseline estimation, so every trajectory must be rolled out to its terminal reward. We introduce Actor-Critic with Action Chunking (AC2) that removes the need to roll every trajectory to completion. AC2 instead assigns credit to action chunks: short continuations of prefixes of past trajectories. A learned critic scores the state reached at the end of each action chunk, allowing the policy to update without observing a terminal reward. We make critic-based credit assignment reliable through three design choices. First, we introduce local readiness which uses critic-based updates on a problem only when the critic is sufficiently accurate on that particular problem. Second, when available, we provide the critic with a reference solution from a previous successful rollout. Third, we assign credit over action chunks of 10k tokens rather than individual tokens, giving the critic a more meaningful portion of the trajectory to evaluate. We train Qwen3-4B on FineProofs-RL using AC2 and evaluate on IMO-ProofBench. AC2 exceeds GRPO's peak validation score of 18.5% using 2.5x fewer decoding FLOPs. This gain comes from two sources, (1) AC2 requires 25% fewer training steps to reach this score, and (2) each step generates fewer tokens because the policy does not need to continue every trajectory to completion. Conceptually, we demonstrate that we can remove the need to roll out every trajectory to completion, opening up a large previously unexplored design space for LLM RL algorithms. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.39247 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T12:14:14.040Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).