Metraipolis › archive › dc_69d87edd44e1
A Sign Law for Entropy Drift under Sampler Mismatch in RLVR — arXiv:2610.34044
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.34044
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-27T08:00:40.280Z
- SHA-256 of the archived text
97b3d5a368b7f2e87f2ad74897dafbb3cc5da3113f254fc1af426663f40d78c8- Stable URL of this page
https://metraipolis.com/documento/dc_69d87edd44e1
How to cite
A Sign Law for Entropy Drift under Sampler Mismatch in RLVR — arXiv:2610.34044 — arXiv (en), https://arxiv.org/abs/2610.34044 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_69d87edd44e1 (Metraipolis archive dc_69d87edd44e1, sha256 97b3d5a368b7f2e87f2ad74897dafbb3cc5da3113f254fc1af426663f40d78c8)
BibTeX
@misc{dc69d87edd44e1,
title = {A Sign Law for Entropy Drift under Sampler Mismatch in RLVR — arXiv:2610.34044},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.34044},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_69d87edd44e1; sha256 97b3d5a368b7f2e87f2ad74897dafbb3cc5da3113f254fc1af426663f40d78c8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.34044. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Entropy in reinforcement learning with verifiable rewards (RLVR) moves up or down depending on choices that should be neutral, such as whether over-length responses are masked or counted as failures. We trace this to sampler mismatch, where rollouts are drawn with top-k/top-p truncation or at a temperature the learner does not apply, and prove that a constant advantage shift then changes the entropy of a softmax policy, at first order and in expectation, with a sign fixed by the sampler: positive shifts sharpen the policy under any sampler whose likelihood ratio to the policy increases with token probability, including top-k, top-p, min-p, and temperatures below one, and flatten it under flatter samplers. In group-based RLVR with token-level losses, the baseline-controlled part of this drift is set by the token-weighted net advantage, which masking and failing over-length responses can push to opposite signs; the remainder, a covariance between advantages and the sampler's sharpness gap, can feed back over long horizons. On a 2B model, the entropy slope is linear in shifts of the net advantage under truncated sampling and indistinguishable from flat under exact sampling, and token-level importance ratios leave 98% of the drift. In 100-step runs, a net-zero baseline only delays an entropy explosion that exact sampling does not show, a fixed positive shift delays it further, and a baseline weighted by the sharpness gap keeps entropy stable without changing the learner. On a 9B reasoning model over ten steps, the uncorrected rule loses held-out accuracy, whereas zeroing the net advantage or renormalizing the learner gains 8.2 and 8.8 points over it, with no held-out effect under exact sampling. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.34044 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-27T07:55:30.873Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).