Metraipolis › archive › dc_4592af28bfef

Entropy Regularization: A Free Correction to Cross-Entropy for Verified Demonstrations — arXiv:2609.30572

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.30572
License (as archived): CC0-1.0
Archived: 2026-10-08T02:10:26.372Z
SHA-256 of the archived text: c74a1a7befd685e3e4a9b1cd0c28aa54a9397c7cd53049a377fb585ca025c35b
Stable URL of this page: https://metraipolis.com/documento/dc_4592af28bfef

How to cite

Entropy Regularization: A Free Correction to Cross-Entropy for Verified Demonstrations — arXiv:2609.30572 — arXiv (en), https://arxiv.org/abs/2609.30572 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4592af28bfef (Metraipolis archive dc_4592af28bfef, sha256 c74a1a7befd685e3e4a9b1cd0c28aa54a9397c7cd53049a377fb585ca025c35b)

BibTeX

@misc{dc4592af28bfef,
  title = {Entropy Regularization: A Free Correction to Cross-Entropy for Verified Demonstrations — arXiv:2609.30572},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.30572},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4592af28bfef; sha256 c74a1a7befd685e3e4a9b1cd0c28aa54a9397c7cd53049a377fb585ca025c35b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.30572. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models are often post-trained on expert demonstrations using cross-entropy (CE), even when the downstream objective is not to imitate the demonstrated solution but to produce any output accepted by a verifier. This mismatch is seen in verifiable domains with multiple correct solutions, such as mathematical reasoning and code generation, where training data may contain only one expert solution per problem. We show that minimizing cross-entropy can be misaligned with minimizing verifier risk; two policies can assign identical likelihood to the observed demonstrations while placing different probability mass on incorrect outputs. This is formalized through a learning-theoretic counterexample in which CE minimization selects a suboptimal policy. We identify that controlling the support of the learned policy can solve this problem by preventing probability mass from spreading to unsupported outputs. Since support size is non-differentiable and computationally intractable, we propose entropy-regularized cross-entropy (ER-CE), using token-level Shannon entropy as a tractable proxy. Finally, across mathematical reasoning and code-generation benchmarks, we find that entropy-regularized training consistently improves verifier accuracy over standard cross-entropy. Our results identify a simple failure mode of imitation-based post-training in verifiable tasks and provide a practical objective that is better aligned with producing correct outputs.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-28 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2609.30572 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T01:43:06.680Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).