Metraipolis › archive › dc_22f712124f88

Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning — arXiv:2609.33781

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.33781
License (as archived)
CC0-1.0
Archived
2026-10-07T21:01:46.002Z
SHA-256 of the archived text
e449aef0df204412c2fa855b80c0f155d1f8f0d51a87f943c6b6465575110e95
Stable URL of this page
https://metraipolis.com/documento/dc_22f712124f88

How to cite

Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning — arXiv:2609.33781 — arXiv (en), https://arxiv.org/abs/2609.33781 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_22f712124f88 (Metraipolis archive dc_22f712124f88, sha256 e449aef0df204412c2fa855b80c0f155d1f8f0d51a87f943c6b6465575110e95)
BibTeX
@misc{dc22f712124f88,
  title = {Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning — arXiv:2609.33781},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.33781},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_22f712124f88; sha256 e449aef0df204412c2fa855b80c0f155d1f8f0d51a87f943c6b6465575110e95},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33781. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning with verifiable rewards (RLVR) enhances reasoning in large language models (LLMs) through outcome-level feedback, yet recent approaches to finer-grained credit assignment often require auxiliary models, additional sampling, or privileged information. Although policy entropy provides a readily available signal, prioritizing uncertain positions under both reinforcement and penalization concentrates penalties where failed responses still retain alternatives for recovery, which can suppress opportunities for exploration. To address this, we introduce Entropic Advantage Policy Optimization (EAPO), an entropy-guided credit assignment method that treats success and failure asymmetrically. Specifically, motivated by the observation that success under uncertainty is less repeatable while confident failures tend to recur, EAPO couples normalized policy entropy with the sign of the response advantage to reinforce surprising success and correct repeated failure. It assigns stronger reinforcement to high-entropy decisions in successful responses and stronger penalties to low-entropy decisions in failed responses, while attenuating penalties at uncertain positions to preserve opportunities for recovery. By redistributing the response advantage across tokens, EAPO derives token-level credit directly from existing rollout signals without additional supervision. We validate EAPO on a range of reasoning tasks across both base and reasoning backbones, demonstrating that it achieves the best overall performance. We further show that EAPO promotes more effective exploration, broadening problem coverage and generating more diverse candidate answers.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.33781 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:02.744Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).