Metraipolis › archive › dc_a92db6e98ea9
Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding — arXiv:2610.07342
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.07342
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T11:18:33.810Z
- SHA-256 of the archived text
87e6a35eda7302d2ea8449e712e837731678ef2ca416be33da3d4d4170251d2c- Stable URL of this page
https://metraipolis.com/documento/dc_a92db6e98ea9
How to cite
Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding — arXiv:2610.07342 — arXiv (en), https://arxiv.org/abs/2610.07342 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a92db6e98ea9 (Metraipolis archive dc_a92db6e98ea9, sha256 87e6a35eda7302d2ea8449e712e837731678ef2ca416be33da3d4d4170251d2c)
BibTeX
@misc{dca92db6e98ea9,
title = {Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding — arXiv:2610.07342},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.07342},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a92db6e98ea9; sha256 87e6a35eda7302d2ea8449e712e837731678ef2ca416be33da3d4d4170251d2c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07342. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
On-policy reinforcement learning has become a central paradigm for improving the reasoning abilities of large language models. However, its effectiveness is often limited by reward sparsity: when a model fails to discover correct trajectories for difficult problems, the optimization process receives little useful signal and may stagnate. Existing approaches mitigate this issue by incorporating off-policy demonstrations, expert traces, or model-generated solutions, but they typically require the auxiliary data to match the format of the reinforcement-learning task, often relying on rejection sampling from stronger models to obtain suitable training trajectories. We introduce Rationale-Guided Policy Optimization (RGPO), a framework that adaptively leverages ground-truth rationale information according to the model's current capability while preserving its freedom to explore. Rather than treating reference solutions as fixed imitation targets, RGPO uses them as temporary scaffolds: rationales help the model generate improved responses, after which only higher-reward, model-generated solutions are transferred back to the original unguided setting. This design allows training to exploit available ground-truth information without requiring off-policy data to follow the same format as the RL task. Across both language-only and vision-language reasoning settings, RGPO consistently improves performance over RLVR baselines, and ablation studies show that adaptive rationale guidance is a key contributor to these gains. These results suggest that RGPO offers a practical and general approach for reducing reward sparsity, stabilizing reinforcement learning, and improving reasoning performance in both text-only and multimodal models. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.07342 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T11:15:24.498Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).