Metraipolis › archive › dc_619431ca08cd
Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO — arXiv:2609.03941
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.03941
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T01:12:10.596Z
- SHA-256 of the archived text
70b8f053b5e37fe37c5e7b5534095f3fe7f919685589eb3a49db50ee2afc57c2- Stable URL of this page
https://metraipolis.com/documento/dc_619431ca08cd
How to cite
Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO — arXiv:2609.03941 — arXiv (en), https://arxiv.org/abs/2609.03941 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_619431ca08cd (Metraipolis archive dc_619431ca08cd, sha256 70b8f053b5e37fe37c5e7b5534095f3fe7f919685589eb3a49db50ee2afc57c2)
BibTeX
@misc{dc619431ca08cd,
title = {Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO — arXiv:2609.03941},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.03941},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_619431ca08cd; sha256 70b8f053b5e37fe37c5e7b5534095f3fe7f919685589eb3a49db50ee2afc57c2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03941. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
RL-based post-training for reasoning models is increasingly bottlenecked by repeated fresh rollout generation, particularly in agentic settings where environment interaction dominates wall-clock cost. Replay can reduce this burden by reusing past trajectories, but existing methods typically embed it within larger training pipelines involving exploration, experience restructuring, or mixed-policy optimization. This makes replay's own contribution difficult to isolate. We ask a focused question: how far can principled replay selection alone go? We introduce Headroom-Drift Replay, a group-level replay control primitive for GRPO that separates reuse into two decisions. Headroom ranks stored groups by remaining learning value, while Drift gates them by compatibility with the current policy. The fresh on-policy stream remains unchanged, and the method adds no auxiliary generation or training machinery. Across mathematical reasoning, multimodal reasoning, and Agentic Search benchmarks, this single intervention outperforms naive replay and matches or exceeds broader replay methods on Avg Mean@32. In Agentic Search, where environment interaction dominates cost, it delivers comparable quality at materially lower wall-clock time. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.03941 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T00:28:03.016Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).