Metraipolis › archive › dc_bfb6b9e708ea
Save Your Saturated Data: Learning Beyond Reward Saturation in Group-Based RL — arXiv:2609.33126
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33126
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:53:58.874Z
- SHA-256 of the archived text
fd9e6e11519d7f1296ca44f8978e7aa5aa4d8d689e143f5c39f84b97e0cf674f- Stable URL of this page
https://metraipolis.com/documento/dc_bfb6b9e708ea
How to cite
Save Your Saturated Data: Learning Beyond Reward Saturation in Group-Based RL — arXiv:2609.33126 — arXiv (en), https://arxiv.org/abs/2609.33126 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bfb6b9e708ea (Metraipolis archive dc_bfb6b9e708ea, sha256 fd9e6e11519d7f1296ca44f8978e7aa5aa4d8d689e143f5c39f84b97e0cf674f)
BibTeX
@misc{dcbfb6b9e708ea,
title = {Save Your Saturated Data: Learning Beyond Reward Saturation in Group-Based RL — arXiv:2609.33126},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33126},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bfb6b9e708ea; sha256 fd9e6e11519d7f1296ca44f8978e7aa5aa4d8d689e143f5c39f84b97e0cf674f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33126. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Group-relative reinforcement learning (RL) relies on reward variation among sampled responses to estimate informative relative advantages. As language models become increasingly capable, existing training data can become reward-saturated: all sampled responses to the same problem might receive equally high rewards, where the group-relative learning signals vanish and leave previously useful data obsolete. In this work, we investigate whether useful learning signals can be recovered from such saturated data. We study interventions at four levels of group-policy RL pipelines---data, rollout, reward, and advantage---and conduct extensive RL training on saturated reasoning data only. While standard GRPO on saturated data would almost always yield near-0 advantages and near-noise signals, diverse interventions successfully recycle and repurpose such data: among the proposed strategies, interventions at rollout generation are consistently most effective: nudging the policy to generate ``high-quality'', incorrect solutions introduces rollouts with poor rewards into saturated groups as negative samples, which turns out to improve GRPO by 6.4% to 9.0% across Qwen3-1.7B and 4B. Other interventions such as increasing rollout temperature or adding auxiliary rewards can also restore non-zero advantages, but yield less consistent gains. Further analyses show that effective negative rollouts require informative negative trajectories, that the method remains effective alongside unsaturated data, and that it supports iterative recycling of newly saturated examples. While increasingly stronger LLMs would render more data as saturated, our results demonstrate that don't waste your saturated data: with the right strategies they can be recycled into useful RL training signals in an increasingly data-scarce world. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.33126 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T21:27:08.750Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).