Metraipolis › archive › dc_0aeec983b6ea

Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling — arXiv:2610.20497

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.20497
License (as archived)
CC0-1.0
Archived
2026-10-15T14:20:38.923Z
SHA-256 of the archived text
d1fc5c856886e983dc18c6af8ef6f9ee1894aea2aa60a9dd06b1a2358be0660d
Stable URL of this page
https://metraipolis.com/documento/dc_0aeec983b6ea

How to cite

Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling — arXiv:2610.20497 — arXiv (en), https://arxiv.org/abs/2610.20497 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0aeec983b6ea (Metraipolis archive dc_0aeec983b6ea, sha256 d1fc5c856886e983dc18c6af8ef6f9ee1894aea2aa60a9dd06b1a2358be0660d)
BibTeX
@misc{dc0aeec983b6ea,
  title = {Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling — arXiv:2610.20497},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.20497},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0aeec983b6ea; sha256 d1fc5c856886e983dc18c6af8ef6f9ee1894aea2aa60a9dd06b1a2358be0660d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20497. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual reasoning tokens or actions remains difficult. Group-relative methods such as Group Relative Policy Optimization (GRPO) sample several attempts per prompt, but lose their reward-based learning signal when all attempts receive the same outcome. We propose Hindsight-Privileged RLVR (HP-RLVR), a single-rollout method that turns a completed trajectory's own future into token-level update weights. The generating policy snapshot scores each eligible token under its causal history and under a hindsight context containing the realized future and verifier feedback. A normalized Jensen--Shannon divergence between the two views weights the reward relative to a historical baseline, yielding bounded token credit for both successful and failed attempts, with no reference solution or learned critic. In paired Qwen3-8B runs over a shared $100$-step schedule, HP-RLVR trains $1.21\times$, $1.46\times$, and $2.30\times$ faster than GRPO on mathematics, code generation, and ALFWorld, using one eighth the rollouts per prompt. Accuracy is comparable overall, with HP-RLVR ahead on mathematics at all three sizes and GRPO ahead on code and ALFWorld. In ALFWorld, $43.2\%$ (4B) and $49.0\%$ (8B) of GRPO's gradient-norm records are zero, versus none for HP-RLVR.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.20497 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T13:53:35.764Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).