Metraipolis › archive › dc_e34139ff76e7
Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization — arXiv:2609.39402
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.39402
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T12:42:46.419Z
- SHA-256 of the archived text
14db1ee0bc7df32cb4e937aedecfce35f7283679064053090da87c1698865f85- Stable URL of this page
https://metraipolis.com/documento/dc_e34139ff76e7
How to cite
Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization — arXiv:2609.39402 — arXiv (en), https://arxiv.org/abs/2609.39402 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e34139ff76e7 (Metraipolis archive dc_e34139ff76e7, sha256 14db1ee0bc7df32cb4e937aedecfce35f7283679064053090da87c1698865f85)
BibTeX
@misc{dce34139ff76e7,
title = {Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization — arXiv:2609.39402},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.39402},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e34139ff76e7; sha256 14db1ee0bc7df32cb4e937aedecfce35f7283679064053090da87c1698865f85},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39402. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Value-model-free RLVR methods such as GRPO assign uniform advantages to all tokens in a rollout, ignoring that tokens contribute unequally. Recent methods use token entropy as an importance proxy but compute it globally across the batch, conflating importance with prompt difficulty and positional trends. We argue that importance should instead be measured relative to the local context of each token. We introduce proximal entropy, a local measure of token importance relative to neighboring tokens, and prove it is invariant to both confounders. Proximal Entropy Policy Optimization (PEPO) uses it to weight per-token advantages and outperforms GRPO and entropy-based baselines on mathematical reasoning across Qwen3-1.7B, Qwen3-4B, and Llama-3.2-3B-Instruct. We also show the formulation generalizes to other algorithms where substituting proximal entropy into existing methods improves, and applying it to single-stream RL succeeds where global entropy fails. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.39402 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T12:14:13.763Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).