Metraipolis › archive › dc_857f62069be4
Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining — arXiv:2609.40089
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.40089
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T13:46:52.889Z
- SHA-256 of the archived text
2f801ec2dbdf071f9953cd53cb324de38afa325235e82ffa91654187c5acd350- Stable URL of this page
https://metraipolis.com/documento/dc_857f62069be4
How to cite
Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining — arXiv:2609.40089 — arXiv (en), https://arxiv.org/abs/2609.40089 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_857f62069be4 (Metraipolis archive dc_857f62069be4, sha256 2f801ec2dbdf071f9953cd53cb324de38afa325235e82ffa91654187c5acd350)
BibTeX
@misc{dc857f62069be4,
title = {Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining — arXiv:2609.40089},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.40089},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_857f62069be4; sha256 2f801ec2dbdf071f9953cd53cb324de38afa325235e82ffa91654187c5acd350},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.40089. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Continued pretraining enables language models to adapt to new domains and knowledge, but often at the cost of forgetting previously acquired capabilities. Replay can mitigate this trade-off, but fixed replay mixtures allocate training independently of the model's actual retention needs. We introduce Replay on Demand (RoD), which instead derives the replay allocation from the model's learning dynamics. RoD jointly prioritizes adaptation samples by their remaining learning potential and replay samples by their observed forgetting. Their competition for a shared training budget yields an online curriculum that determines what to train on at each step. Across models, scales, and adaptation domains, RoD reaches or improves upon the adaptation-forgetting frontier of tuned fixed-replay baselines and model merging without prescribing a replay allocation in advance. Replay concentrates on sources that are more vulnerable to forgetting and dynamically increases and redistributes as forgetting emerges during training. Together, our results show that replay can be allocated online from the model's evolving state, targeting what is needed, when it is needed. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.40089 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T13:18:10.503Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).