Metraipolis › archive › dc_ae753ae6ba2d

LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates — arXiv:2609.02734

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.02734
License (as archived)
CC0-1.0
Archived
2026-10-17T05:00:43.894Z
SHA-256 of the archived text
a6e265e3b2d9256d64056b64ed9286ee92337b32e53aacde0b38670a198d3976
Stable URL of this page
https://metraipolis.com/documento/dc_ae753ae6ba2d

How to cite

LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates — arXiv:2609.02734 — arXiv (en), https://arxiv.org/abs/2609.02734 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ae753ae6ba2d (Metraipolis archive dc_ae753ae6ba2d, sha256 a6e265e3b2d9256d64056b64ed9286ee92337b32e53aacde0b38670a198d3976)
BibTeX
@misc{dcae753ae6ba2d,
  title = {LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates — arXiv:2609.02734},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.02734},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ae753ae6ba2d; sha256 a6e265e3b2d9256d64056b64ed9286ee92337b32e53aacde0b38670a198d3976},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.02734. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Low-rank adaptation (LoRA) is the standard way to fine-tune large models, yet when its two factors are trained independently, the update ignores the geometry of the low-rank weight change it induces. We introduce LoRA-TSD, an optimizer that treats every LoRA step as a tangent vector of the fixed-rank matrix manifold and takes the spectral-norm steepest-descent step of Muon inside that tangent space, mapping the result back to the factors through a retraction native to the LoRA parametrization. The step avoids expensive operations on full weight matrices, and its retraction is up to $2.8\times$ cheaper than the truncated-SVD retraction used by prior manifold methods. We prove that the Frobenius-norm version of our surrogate recovers LoRA-Pro, and we identify the tangent-projected gradient, the Riemannian gradient of the manifold, as the stationarity measure natural to LoRA training and computable from the factor gradients alone. Under this measure we give the first global convergence guarantees for both LoRA-Pro and LoRA-TSD, with rates that drive the factor-gradient norms to zero. Across six commonsense and natural-language-inference benchmarks with Llama-3.2-1B, Llama-3.1-8B and Qwen3-32B, LoRA-TSD outperforms every competing LoRA optimizer and stays robust to the adapter rank. Code is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-03 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.02734 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T04:09:43.335Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-03?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).