Metraipolis › archive › dc_98af5b50aebe
RL-finetuning LLMs from on- and off-policy data with a single algorithm — arXiv:2503.19612
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2503.19612
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T09:32:48.496Z
- SHA-256 of the archived text
d95c31c5bc86eaa27d0c3c1f8d648034cc37cef773ab6c301da35a739c8f05e0- Stable URL of this page
https://metraipolis.com/documento/dc_98af5b50aebe
How to cite
RL-finetuning LLMs from on- and off-policy data with a single algorithm — arXiv:2503.19612 — arXiv (en), https://arxiv.org/abs/2503.19612 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_98af5b50aebe (Metraipolis archive dc_98af5b50aebe, sha256 d95c31c5bc86eaa27d0c3c1f8d648034cc37cef773ab6c301da35a739c8f05e0)
BibTeX
@misc{dc98af5b50aebe,
title = {RL-finetuning LLMs from on- and off-policy data with a single algorithm — arXiv:2503.19612},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2503.19612},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_98af5b50aebe; sha256 d95c31c5bc86eaa27d0c3c1f8d648034cc37cef773ab6c301da35a739c8f05e0},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2503.19612. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We introduce a novel reinforcement learning algorithm (AGRO, for Any-Generation Reward Optimization) for fine-tuning large-language models. AGRO leverages the concept of generation consistency, which states that the optimal policy satisfies the notion of consistency across any possible generation of the model. We derive algorithms that find optimal solutions via the sample-based policy gradient and provide theoretical guarantees on their convergence. Our experiments demonstrate the effectiveness of AGRO in both on-policy and off-policy settings, showing improved performance on the mathematical reasoning dataset over baseline algorithms. Tema degli esploratori: statistica e scienza dei dati (elenco arXiv stat.ML, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML). Pagina del lavoro: https://arxiv.org/abs/2503.19612 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T08:24:13.959Z dalla pagina https://arxiv.org/list/stat.ML/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).