Metraipolis › archive › dc_0e5da68fac4b

Continual Learning Mechanisms Compose for Long-Horizon Memorization — arXiv:2609.06986

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.06986
License (as archived)
CC0-1.0
Archived
2026-10-16T13:57:36.674Z
SHA-256 of the archived text
7fb2fab64a9bac23a8c1f962c9c2d45b6f3ebeea853765ef110f0b53a38d76e3
Stable URL of this page
https://metraipolis.com/documento/dc_0e5da68fac4b

How to cite

Continual Learning Mechanisms Compose for Long-Horizon Memorization — arXiv:2609.06986 — arXiv (en), https://arxiv.org/abs/2609.06986 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0e5da68fac4b (Metraipolis archive dc_0e5da68fac4b, sha256 7fb2fab64a9bac23a8c1f962c9c2d45b6f3ebeea853765ef110f0b53a38d76e3)
BibTeX
@misc{dc0e5da68fac4b,
  title = {Continual Learning Mechanisms Compose for Long-Horizon Memorization — arXiv:2609.06986},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.06986},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0e5da68fac4b; sha256 7fb2fab64a9bac23a8c1f962c9c2d45b6f3ebeea853765ef110f0b53a38d76e3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.06986. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Language models may need to internalize information that arrives over time and retain it through many subsequent updates. To study this challenge, we introduce long-horizon memorization, a setting in which a model learns 100 query-answer tasks through continual supervised fine-tuning without retaining earlier training examples or receiving task identifiers at inference. Sequential updates cause catastrophic forgetting, and no single continual learning mechanism we evaluate maintains strong retention at this horizon. We hypothesize that mechanisms addressing complementary sources of forgetting will be more effective when composed. We organize these compositions along two design dimensions. Data, function, and weight anchors specify what prior information each update should preserve, while low-rank allocation rules determine where successive updates are retained. To test this hypothesis systematically, we construct three distinct 100-task memorization datasets. We introduce task-level successive halving to search the combinatorial design space and use a factorial experiment to measure individual and interaction effects. Our best method combines all three anchors with merged LoRA, ranks among the top 3 methods in all datasets, and raises average final retention from 1.2% under naive sequential fine-tuning to 34.9%, a 28-fold improvement. The data anchor and merged LoRA provide the largest average gains and interact super-additively on all three datasets. Together, these results show that composing complementary mechanisms substantially improves long-horizon memorization beyond what any individual mechanism achieves.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.06986 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T13:39:42.926Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).