Metraipolis › archive › dc_d19657c0f372

ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards — arXiv:2609.36058

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.36058
License (as archived)
CC0-1.0
Archived
2026-10-07T17:28:39.921Z
SHA-256 of the archived text
038705081d9dcaf255cebcf8033aa6b379ce4cb26daef9782d568cefc0d34eee
Stable URL of this page
https://metraipolis.com/documento/dc_d19657c0f372

How to cite

ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards — arXiv:2609.36058 — arXiv (en), https://arxiv.org/abs/2609.36058 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d19657c0f372 (Metraipolis archive dc_d19657c0f372, sha256 038705081d9dcaf255cebcf8033aa6b379ce4cb26daef9782d568cefc0d34eee)
BibTeX
@misc{dcd19657c0f372,
  title = {ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards — arXiv:2609.36058},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.36058},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d19657c0f372; sha256 038705081d9dcaf255cebcf8033aa6b379ce4cb26daef9782d568cefc0d34eee},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36058. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recent progress in reinforcement learning with verifiable rewards (RLVR) has highlighted the effectiveness of simple critic-free policy-gradient methods such as Group Relative Policy Optimization (GRPO). In contrast, actor-critic methods rely on learned value functions whose approximation error can introduce bias through commonly used advantage estimators such as temporal-difference error. Motivated by this observation, we revisit trajectory-level control variates through an advantage-value formulation, which we call Advantage-Based Control Variates (ABC). This formulation reveals that the covariance structure is closely related to the return decomposition used in Direct Advantage Estimation (DAE). Finally, we combine ABC with DAE into a single actor-critic algorithm and evaluate it in an offline-to-online RLVR setting, where the critic is first trained on previously collected trajectories and adapted during online learning. On mathematical reasoning tasks, ABC achieves performance competitive with GRPO using substantially fewer online optimization steps.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.36058 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T16:59:48.094Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).