Metraipolis › archive › dc_0aeec983b6ea
Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling — arXiv:2610.20497
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.20497
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T14:20:38.923Z
- SHA-256 of the archived text
d1fc5c856886e983dc18c6af8ef6f9ee1894aea2aa60a9dd06b1a2358be0660d- Stable URL of this page
https://metraipolis.com/documento/dc_0aeec983b6ea
How to cite
Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling — arXiv:2610.20497 — arXiv (en), https://arxiv.org/abs/2610.20497 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0aeec983b6ea (Metraipolis archive dc_0aeec983b6ea, sha256 d1fc5c856886e983dc18c6af8ef6f9ee1894aea2aa60a9dd06b1a2358be0660d)
BibTeX
@misc{dc0aeec983b6ea,
title = {Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling — arXiv:2610.20497},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.20497},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0aeec983b6ea; sha256 d1fc5c856886e983dc18c6af8ef6f9ee1894aea2aa60a9dd06b1a2358be0660d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20497. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual reasoning tokens or actions remains difficult. Group-relative methods such as Group Relative Policy Optimization (GRPO) sample several attempts per prompt, but lose their reward-based learning signal when all attempts receive the same outcome. We propose Hindsight-Privileged RLVR (HP-RLVR), a single-rollout method that turns a completed trajectory's own future into token-level update weights. The generating policy snapshot scores each eligible token under its causal history and under a hindsight context containing the realized future and verifier feedback. A normalized Jensen--Shannon divergence between the two views weights the reward relative to a historical baseline, yielding bounded token credit for both successful and failed attempts, with no reference solution or learned critic. In paired Qwen3-8B runs over a shared $100$-step schedule, HP-RLVR trains $1.21\times$, $1.46\times$, and $2.30\times$ faster than GRPO on mathematics, code generation, and ALFWorld, using one eighth the rollouts per prompt. Accuracy is comparable overall, with HP-RLVR ahead on mathematics at all three sizes and GRPO ahead on code and ALFWorld. In ALFWorld, $43.2\%$ (4B) and $49.0\%$ (8B) of GRPO's gradient-norm records are zero, versus none for HP-RLVR. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.20497 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T13:53:35.764Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).