Metraipolis › archive › dc_35ec3d547005

Boosting LLM Exploration via Weak-Model Guidance in RLVR — arXiv:2608.27420

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.27420
License (as archived)
CC0-1.0
Archived
2026-10-17T15:20:45.862Z
SHA-256 of the archived text
9d67827acc681d1da6d50069d8c12562c0e377539bf30002918c91eead27221f
Stable URL of this page
https://metraipolis.com/documento/dc_35ec3d547005

How to cite

Boosting LLM Exploration via Weak-Model Guidance in RLVR — arXiv:2608.27420 — arXiv (en), https://arxiv.org/abs/2608.27420 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_35ec3d547005 (Metraipolis archive dc_35ec3d547005, sha256 9d67827acc681d1da6d50069d8c12562c0e377539bf30002918c91eead27221f)
BibTeX
@misc{dc35ec3d547005,
  title = {Boosting LLM Exploration via Weak-Model Guidance in RLVR — arXiv:2608.27420},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.27420},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_35ec3d547005; sha256 9d67827acc681d1da6d50069d8c12562c0e377539bf30002918c91eead27221f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27420. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@$k$ for large $k$. While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically study the potential of outer prefixes, revealing the mechanism of the impact of distributional discrepancy to the exploration dynamics in RLVR training. Experiments across multiple mathematical benchmarks show that our method consistently outperforms vanilla RLVR. Notably, the performance gain becomes increasingly pronounced as $k$ scales up, demonstrating a substantial expansion of reasoning coverage. Furthermore, our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.27420 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T15:15:18.101Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).