Metraipolis › archive › dc_5a770fb80fcd

Correlation-Aware Exploration for Long-Trace Reasoning — arXiv:2610.34949

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.34949
License (as archived): CC0-1.0
Archived: 2026-10-27T08:02:25.250Z
SHA-256 of the archived text: aa4160e10f108eca0667367e6c81ece21c5e824c0148f022a4635220d245a164
Stable URL of this page: https://metraipolis.com/documento/dc_5a770fb80fcd

How to cite

Correlation-Aware Exploration for Long-Trace Reasoning — arXiv:2610.34949 — arXiv (en), https://arxiv.org/abs/2610.34949 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5a770fb80fcd (Metraipolis archive dc_5a770fb80fcd, sha256 aa4160e10f108eca0667367e6c81ece21c5e824c0148f022a4635220d245a164)

BibTeX

@misc{dc5a770fb80fcd,
  title = {Correlation-Aware Exploration for Long-Trace Reasoning — arXiv:2610.34949},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.34949},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5a770fb80fcd; sha256 aa4160e10f108eca0667367e6c81ece21c5e824c0148f022a4635220d245a164},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.34949. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reasoning models trained with Group Relative Policy Optimization (GRPO) now produce chains of thought thousands of tokens long, and recent studies reveal that such training is prone to policy collapse. Entropy regularization is the standard remedy, but it works by pushing probability mass away from the few tokens that carry it and toward the vast low-probability tail of the vocabulary, and it applies this force once for every token of the reasoning trace. This puts the bonus in direct conflict with the entropy decay that convergence requires, and makes its coefficient difficult to set in the long-trace regime. To resolve this tension, we propose Info-GRPO, an information-theoretic framework that cultivates correlation between the policy and a latent prior. Rather than raising entropy everywhere, Info-GRPO builds on the mutual information between a latent variable and the trajectory it induces: augmenting prompts with latent seeds during training lets the model explore diverse policies correlated with the prior, while the seed-conditioned entropy is guided toward convergence. Diversity is therefore carried across whole reasoning traces rather than by per-token randomness, and exploration no longer fights entropy reduction. Experiments show that Info-GRPO outperforms vanilla GRPO and entropy-regularized GRPO across diverse reasoning benchmarks. On AIME24, it improves $Avg@8$ over GRPO by 3.75%, 1.66%, 4.16%, and 4.17% on four backbones, and it reaches the same majority-vote accuracy with $8\times$ fewer samples. Code and models will be publicly released.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.34949 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:55:30.953Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).