Metraipolis › archive › dc_f97ec9f0935b

Muon Can Outperform Dedicated Continual Learning Methods — arXiv:2609.24678

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.24678
License (as archived)
CC0-1.0
Archived
2026-10-09T06:50:42.863Z
SHA-256 of the archived text
f7b3704c11f964be9df702ac616fc8b313249bf1213922a93afc51ab52df39af
Stable URL of this page
https://metraipolis.com/documento/dc_f97ec9f0935b

How to cite

Muon Can Outperform Dedicated Continual Learning Methods — arXiv:2609.24678 — arXiv (en), https://arxiv.org/abs/2609.24678 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f97ec9f0935b (Metraipolis archive dc_f97ec9f0935b, sha256 f7b3704c11f964be9df702ac616fc8b313249bf1213922a93afc51ab52df39af)
BibTeX
@misc{dcf97ec9f0935b,
  title = {Muon Can Outperform Dedicated Continual Learning Methods — arXiv:2609.24678},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.24678},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f97ec9f0935b; sha256 f7b3704c11f964be9df702ac616fc8b313249bf1213922a93afc51ab52df39af},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.24678. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Continual learning with Low-Rank Adapters (LoRA) typically mitigates forgetting by penalizing the overlap between a new update and the accumulated past weights, which discourages certain update directions without controlling how an update distributes its energy over the ones that remain. We ask whether that restriction has to be task-aware, or whether a generic one supplied by the optimizer is enough. We train a plain incremental LoRA (IncLoRA) with Muon, which orthogonalizes each update, and compare it against O-LoRA and ELLA over five seeds and three task orders on the Standard CL Benchmark and three seeds on TRACE. IncLoRA+Muon reaches the accuracy band of the dedicated methods on Standard CL and improves on every AdamW configuration on TRACE. One update-constraining mechanism is enough, whether it comes from the loss or from the optimizer; on Standard CL a second one does not help, and for the most restrictive method it costs 8.4 points of accuracy and the plasticity to fit each task. What separates the two optimizers is not the size of the update, which under Muon is 0.91 to 2.06 times that under AdamW, but how it is distributed. AdamW confines it to between 1.4 and 1.8 effective singular directions, Muon spreads it over 7.0, and the two do not overlap in any tracked run. Part of the advantage usually attributed to dedicated CL methods may therefore be explained by the geometry of the optimizer's updates.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.24678 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:48:45.153Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).