Metraipolis › archive › dc_ea98e3a2a421

Decoupling Exploration from Optimization in RLVR — arXiv:2610.10536

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.10536
License (as archived)
CC0-1.0
Archived
2026-10-08T11:12:13.228Z
SHA-256 of the archived text
bcfbc5ac7605f2b56bb271c4dcd9f5374fc72f97c319f11e99f49c13f22ac0c6
Stable URL of this page
https://metraipolis.com/documento/dc_ea98e3a2a421

How to cite

Decoupling Exploration from Optimization in RLVR — arXiv:2610.10536 — arXiv (en), https://arxiv.org/abs/2610.10536 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ea98e3a2a421 (Metraipolis archive dc_ea98e3a2a421, sha256 bcfbc5ac7605f2b56bb271c4dcd9f5374fc72f97c319f11e99f49c13f22ac0c6)
BibTeX
@misc{dcea98e3a2a421,
  title = {Decoupling Exploration from Optimization in RLVR — arXiv:2610.10536},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.10536},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ea98e3a2a421; sha256 bcfbc5ac7605f2b56bb271c4dcd9f5374fc72f97c319f11e99f49c13f22ac0c6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.10536. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas absent from its prior training data. In practice, however, augmenting RLVR with strong novelty incentives has seen limited success and can degrade model quality. Because verifiable rewards supervise only a narrow slice of the model's knowledge and behavior, such degradations are difficult to recover from. Instead, we decouple exploration from optimization in a framework we call Exploration-Distillation (ExpDis). We train one or more explorer policies with a novelty bonus in the reward, filter their trajectories for correctness and quality, and distill them into a separate student policy. The student policy is then trained without a novelty bonus. We repeat the above procedure for several rounds, alternating between exploration and optimization. This decoupling allows us to aggressively scale exploration without degrading the student policy. Across seven mathematical reasoning benchmarks and two model families, ExpDis outperforms DAPO at the same wall-clock budget. Moreover, we observe improved pass@$k$ scaling, indicating that ExpDis produces models that generate more diverse correct solutions.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.10536 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:14:06.190Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).