Metraipolis › archive › dc_a38fb52f2dde

When to Review: Spaced Repetition for Continual Pre-Training of Language Models — arXiv:2608.17530

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.17530
License (as archived)
CC0-1.0
Archived
2026-10-18T14:28:43.338Z
SHA-256 of the archived text
4b806ec9750e958d884477cf79527be86f8a945255fe0b417d9e5656ed553ef9
Stable URL of this page
https://metraipolis.com/documento/dc_a38fb52f2dde

How to cite

When to Review: Spaced Repetition for Continual Pre-Training of Language Models — arXiv:2608.17530 — arXiv (en), https://arxiv.org/abs/2608.17530 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a38fb52f2dde (Metraipolis archive dc_a38fb52f2dde, sha256 4b806ec9750e958d884477cf79527be86f8a945255fe0b417d9e5656ed553ef9)
BibTeX
@misc{dca38fb52f2dde,
  title = {When to Review: Spaced Repetition for Continual Pre-Training of Language Models — arXiv:2608.17530},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.17530},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a38fb52f2dde; sha256 4b806ec9750e958d884477cf79527be86f8a945255fe0b417d9e5656ed553ef9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.17530. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Continual pre-training of large language models must acquire new information without erasing old knowledge. Existing replay methods often choose a global old/new mixture and sample uniformly, ignoring that examples differ in how quickly they are forgotten. We formulate continual pre-training as adaptive review scheduling: the training loop should decide not only how much history to replay, but which examples should return at each step. We introduce Spaced Repetition Training (SRT), a continual learning framework inspired by cognitive science, which schedules sample-rehearsal using the SuperMemo-2 (SM-2) algorithm. SRT maintains per-example review state, maps per-example perplexity to a recall-quality signal, and schedules historical examples for retention and new examples for consolidation while leaving the model, objective, and optimizer unchanged. On temporally separated Wikipedia and code corpora, SRT improves the stability-plasticity trade-off, recovering 5 to 37 percentage points of old-knowledge accuracy lost by naive continual pre-training across model scales while preserving or improving new-knowledge acquisition. At larger scale, SRT preserves broad benchmark performance that naive continual pre-training and uniform replay substantially degrade. Experiments with vision and tabular data further suggest that the scheduling principle extends beyond language when paired with an appropriate recall signal.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-19 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.17530 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T14:27:54.274Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-19?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).