Metraipolis › archive › dc_ff850744db63

When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging — arXiv:2608.27518

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.27518
License (as archived)
CC0-1.0
Archived
2026-10-17T11:06:22.152Z
SHA-256 of the archived text
fe34a55f91495b508126713d07d6c87cd1dd00a5f28b25a9e145d0b5f29adfd0
Stable URL of this page
https://metraipolis.com/documento/dc_ff850744db63

How to cite

When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging — arXiv:2608.27518 — arXiv (en), https://arxiv.org/abs/2608.27518 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ff850744db63 (Metraipolis archive dc_ff850744db63, sha256 fe34a55f91495b508126713d07d6c87cd1dd00a5f28b25a9e145d0b5f29adfd0)
BibTeX
@misc{dcff850744db63,
  title = {When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging — arXiv:2608.27518},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.27518},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ff850744db63; sha256 fe34a55f91495b508126713d07d6c87cd1dd00a5f28b25a9e145d0b5f29adfd0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27518. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Continual learning (CL) and model merging (MM) both aim to obtain a single model that performs well across multiple tasks, challenged respectively by catastrophic forgetting and weight-disentanglement error. In the literature, these difficulties are merely treated separately and mitigated through a variety of solutions, while the geometry induced by the base optimizer is treated as an implementation detail. In this work, we show that the two difficulties are in fact two instances of the same phenomenon: a parameter update useful for one task shifts the model's outputs on another. We formalize this shared phenomenon as \textit{task interference} and reduce it to a common layer-wise Frobenius inner product $\langle \Delta W_\ell, J_\ell(x)\rangle_F$. This quantity, in turn, is utilized to expose the role of the optimizer. We theoretically derive an upper bound that isolates the spectral norm $\|\Delta W_\ell\|_2$ as an optimizer-controllable factor of task interference, and a per-mode analysis shows that this bound tracks the dominant part of the empirical interference. Specifically, we then identify the recent Muon optimizer as a mechanism that regulates this factor by construction. Our work reveals that its elegant control on spectral norm tightens the interference bound for both CL and MM, positioning Muon as a principled optimizer-centric approach complementary to existing solutions. Our theoretcal analysis is well validated by experimental results. Replacing the AdamW optimizer with Muon improves accuracy by up to +5.02 points on the eight-task model-merging benchmark across three CLIP backbones. For continual learning, Muon also delivers uniformly positive gains across ten class-incremental protocols, three task-incremental protocols, and the 11-task MTIL benchmark.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-31 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.27518 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T10:50:25.761Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).