Metraipolis › archive › dc_7aa188a19035
Not All Entropy Is Equal: Reward-Conditioned Entropy Allocation in Reinforcement Learning with Verifiable Rewards — arXiv:2611.09448
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.09448
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-09T22:35:04.461Z
- SHA-256 of the archived text
5a245f1439746a0462e357350f836b58464e0a3c1696112d55ff66b9b60b7b23- Stable URL of this page
https://metraipolis.com/documento/dc_7aa188a19035
How to cite
Not All Entropy Is Equal: Reward-Conditioned Entropy Allocation in Reinforcement Learning with Verifiable Rewards — arXiv:2611.09448 — arXiv (en), https://arxiv.org/abs/2611.09448 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7aa188a19035 (Metraipolis archive dc_7aa188a19035, sha256 5a245f1439746a0462e357350f836b58464e0a3c1696112d55ff66b9b60b7b23)
BibTeX
@misc{dc7aa188a19035,
title = {Not All Entropy Is Equal: Reward-Conditioned Entropy Allocation in Reinforcement Learning with Verifiable Rewards — arXiv:2611.09448},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.09448},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7aa188a19035; sha256 5a245f1439746a0462e357350f836b58464e0a3c1696112d55ff66b9b60b7b23},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.09448. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Policy entropy is widely used to diagnose exploration and collapse in reinforcement learning with verifiable rewards (RLVR), yet its magnitude does not consistently track reasoning performance across optimization objectives. We argue that this ambiguity arises because global token entropy conflates uncertainty in successful and failed trajectories, diversity across correct solution strategies, and variation among realizations of the same strategy. We introduce \textbf{Reward-Conditioned Entropy Allocation (RCEA)}, a hierarchical diagnostic framework that separates complete-trajectory entropy from average token entropy, conditions uncertainty on verifier outcome and process validity, measures correct-solution-mode retention, and connects these population-level quantities to local covariance-driven updates. In controlled sweeps over six RLVR objectives at two model scales, entropy trajectories do not induce a stable ordering of reasoning accuracy. Token entropy is higher for verifier-failed than successful trajectories under both broad separately aggregated estimates and a common-question paired analysis, whose confidence intervals exclude equality at 10 of 12 displayed endpoints. A local covariance approximation predicts the direction of adjacent-checkpoint entropy change in 95.5\% of nonfailed transitions; a reward-stratified endpoint proxy further shows that positive chosen-token covariance is more tail-concentrated within failed prefixes. Every reportable trained endpoint exhibits some contraction in independently audited correct-solution-mode breadth, whose positive association with large-\(k\) coverage remains exploratory. These results show that entropy collapse should be diagnosed by which uncertainty is removed, rather than by global entropy alone.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.09448 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-09T22:30:47.330Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).