Metraipolis › archive › dc_fbcad65170c7

Measuring and Mitigating Solution Mode Collapse in RLVR — arXiv:2610.11064

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.11064
License (as archived)
CC0-1.0
Archived
2026-10-09T06:13:44.222Z
SHA-256 of the archived text
838a34b9b10871d83f86d5c28206af94c8f9be99a3412bcab09955a43ca2bed4
Stable URL of this page
https://metraipolis.com/documento/dc_fbcad65170c7

How to cite

Measuring and Mitigating Solution Mode Collapse in RLVR — arXiv:2610.11064 — arXiv (en), https://arxiv.org/abs/2610.11064 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fbcad65170c7 (Metraipolis archive dc_fbcad65170c7, sha256 838a34b9b10871d83f86d5c28206af94c8f9be99a3412bcab09955a43ca2bed4)
BibTeX
@misc{dcfbcad65170c7,
  title = {Measuring and Mitigating Solution Mode Collapse in RLVR — arXiv:2610.11064},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.11064},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fbcad65170c7; sha256 838a34b9b10871d83f86d5c28206af94c8f9be99a3412bcab09955a43ca2bed4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11064. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A language model (LM) can usually answer the same question in more than one way, but reinforcement learning with verifiable rewards (RLVR) is indifferent to which correct answer a model produces. A solution will earn the same reward whether it is the thousandth copy of a familiar answer or one the model has never produced before. Yet, there is potential value in having the model retain multiple correct solutions as it is trained. For instance, multiple modes may give users a choice and provide problem-solving strategies that improve overall model performance. Here, we introduce ModeBench, a benchmark of multi-solution tasks in which the verifier returns both correctness and mode discovered. We then use ModeBench to measure how solution diversity changes under RLVR post-training. We find that RLVR post-training concentrates probability onto fewer correct modes even as accuracy holds or improves, and moreover, that frontier models are already highly concentrated. We then introduce our solution, Re:Max, which stores one verified example per discovered mode in a replay buffer and trains on those stored modes uniformly. A solution found once is, therefore, practiced as often as one found repeatedly. Across three model scales, two RL objectives, and harder task constructions, replay improves both how often a policy succeeds and how many different ways it can succeed.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.11064 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T05:47:15.415Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).