Metraipolis › archive › dc_0a3261e64aec
Towards Continual Learning: Mitigating Forgetting With Subspace Tuning — arXiv:2610.22293
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.22293
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T09:31:09.041Z
- SHA-256 of the archived text
6abfbd6fd485d7e34a952ba56fd3c8fe53db85955c9fe96945bc2c1be13b9ac2- Stable URL of this page
https://metraipolis.com/documento/dc_0a3261e64aec
How to cite
Towards Continual Learning: Mitigating Forgetting With Subspace Tuning — arXiv:2610.22293 — arXiv (en), https://arxiv.org/abs/2610.22293 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0a3261e64aec (Metraipolis archive dc_0a3261e64aec, sha256 6abfbd6fd485d7e34a952ba56fd3c8fe53db85955c9fe96945bc2c1be13b9ac2)
BibTeX
@misc{dc0a3261e64aec,
title = {Towards Continual Learning: Mitigating Forgetting With Subspace Tuning — arXiv:2610.22293},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.22293},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0a3261e64aec; sha256 6abfbd6fd485d7e34a952ba56fd3c8fe53db85955c9fe96945bc2c1be13b9ac2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22293. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) acquire new skills through fine-tuning, but it can destroy capabilities they already have: catastrophic forgetting remains a central obstacle to continual learning in LLMs. Recent works connect forgetting to how far, and along which directions, fine-tuning moves the weights. On-policy methods such as reinforcement learning and on-policy distillation avoid updating the principal singular directions of weight matrices and thus forget less. In contrast, SFT concentrates updates along these directions and is susceptible to catastrophic forgetting. In this work, we introduce Subspace Tuning (SST), which restricts each weight update to the bottom singular subspace, eliminating changes along the principal directions. SST is optimizer-agnostic and composes well with fine-tuning recipes including SFT and self-distillation. SST combines tight subspace protection, dense gradient estimation, and a simple refresh mechanism, enabling stronger learning than prior orthogonal fine-tuning methods. With these mechanisms, protecting as much as 99\% of the weight subspace substantially improves retention while preserving strong target-task performance. This behavior strengthens with model and data size and nearly eliminates forgetting in the large-model regime. SST also keeps the model close to its base distribution, inducing $2$--$4\times$ less KL drift than the baseline recipes. Overall, SST advances the learning--forgetting Pareto frontier, making it a practical default for fine-tuning pre-trained language models. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.22293 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T08:44:18.050Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).