Metraipolis › archive › dc_aeae7cfd9f81

Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation? — arXiv:2608.12332

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.12332
License (as archived)
CC0-1.0
Archived
2026-10-19T04:24:22.409Z
SHA-256 of the archived text
887e802e193d003d3cb37b0fd441b59b883e14f7d05c624c009da66bd6595570
Stable URL of this page
https://metraipolis.com/documento/dc_aeae7cfd9f81

How to cite

Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation? — arXiv:2608.12332 — arXiv (en), https://arxiv.org/abs/2608.12332 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aeae7cfd9f81 (Metraipolis archive dc_aeae7cfd9f81, sha256 887e802e193d003d3cb37b0fd441b59b883e14f7d05c624c009da66bd6595570)
BibTeX
@misc{dcaeae7cfd9f81,
  title = {Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation? — arXiv:2608.12332},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.12332},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aeae7cfd9f81; sha256 887e802e193d003d3cb37b0fd441b59b883e14f7d05c624c009da66bd6595570},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12332. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In recent years, low-rank adaptation (LoRA) has emerged as a significant paradigm that freezes pre-trained weights and introduces small, learnable adapters instead of fine-tuning the full set of parameters. In this work, we uncover several key insights regarding the singular components of network parameters based on Singular Value Decomposition (SVD). Firstly, the principal singular components with large singular values in pre-trained network parameters can be effectively reused during fine-tuning, whereas the minor components with smaller singular values are more task-specific and require substantial adaptation. Secondly, we first establish the theoretical connection that the uncontrolled growth of singular values in LoRA adapters leads to the forgetting of pre-trained knowledge -- a well-known issue referred to as catastrophic forgetting. Building on these observations, we propose SCLoRA, which injects parameterized singular components with spectral clipping into the pre-trained model in a way that is aware of the spectral distribution of the pre-trained model. SCLoRA effectively adapts to new tasks by focusing updates on components that require adaptation, while simultaneously alleviating catastrophic forgetting. We conduct extensive experiments and demonstrate that SCLoRA not only improves downstream performance but also effectively retains pre-trained knowledge.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-14 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.12332 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T04:17:42.362Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).