Metraipolis › archive › dc_20d02f14bbab
Modulating Merging Strengths via Joint Loss Estimation for LoRA-based Continual Learning — arXiv:2611.04931
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.04931
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-04T12:41:00.496Z
- SHA-256 of the archived text
848e9ac3cf5850d1b3d74849673bcb1c3ef8b5f9ee3622a102e50c5c74e3e8e9- Stable URL of this page
https://metraipolis.com/documento/dc_20d02f14bbab
How to cite
Modulating Merging Strengths via Joint Loss Estimation for LoRA-based Continual Learning — arXiv:2611.04931 — arXiv (en), https://arxiv.org/abs/2611.04931 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_20d02f14bbab (Metraipolis archive dc_20d02f14bbab, sha256 848e9ac3cf5850d1b3d74849673bcb1c3ef8b5f9ee3622a102e50c5c74e3e8e9)
BibTeX
@misc{dc20d02f14bbab,
title = {Modulating Merging Strengths via Joint Loss Estimation for LoRA-based Continual Learning — arXiv:2611.04931},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.04931},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_20d02f14bbab; sha256 848e9ac3cf5850d1b3d74849673bcb1c3ef8b5f9ee3622a102e50c5c74e3e8e9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04931. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Continual learning (CL) aims to adapt models to new tasks while preserving previously learned knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL due to its efficiency and scalability. Several LoRA-based merging methods maintain a single inference model by scaling the newly learned LoRA update with one coefficient and merging it into the previous weights after each task. However, they usually control the whole update with a shared scalar factor, although different update elements may affect previous and current tasks in different ways. To address this issue, we propose Merging Strength Modulation via Joint Loss Estimation (M2LE). It consists of Hessian Information-guided Adaptive Merging (HIM) and Task Simulation Perturbation for Merging (TSP). Instead of searching for a global merge coefficient, HIM directly solves for the target merged update under a joint objective over previous and current tasks. Solving this objective gives an analytic solution that naturally takes the form of element-wise modulation over the current task LoRA update. To improve the stability of the learned update under subsequent merging, we introduce TSP which simulates different tasks interference. Experiments on multiple benchmarks show that M2LE consistently outperforms existing PEFT-based continual learning methods. Code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.04931 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-04T12:29:47.314Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).