Metraipolis › archive › dc_b03403f91df5

Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature — arXiv:2609.12123

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.12123
License (as archived)
CC0-1.0
Archived
2026-10-16T03:15:32.046Z
SHA-256 of the archived text
d76c12382b5c0675a40cd2a7f88545a7e7b981f7b078181fd88fd25da3755117
Stable URL of this page
https://metraipolis.com/documento/dc_b03403f91df5

How to cite

Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature — arXiv:2609.12123 — arXiv (en), https://arxiv.org/abs/2609.12123 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b03403f91df5 (Metraipolis archive dc_b03403f91df5, sha256 d76c12382b5c0675a40cd2a7f88545a7e7b981f7b078181fd88fd25da3755117)
BibTeX
@misc{dcb03403f91df5,
  title = {Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature — arXiv:2609.12123},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.12123},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b03403f91df5; sha256 d76c12382b5c0675a40cd2a7f88545a7e7b981f7b078181fd88fd25da3755117},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.12123. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Low-Rank Adaptation (LoRA) is an effective approach for adapting large pretrained models by learning low-rank weight updates. In practice, the LoRA rank is used to control an adapter's parameter budget and representational capacity. We show that this view is incomplete: while the nominal rank determines the representational capacity, the optimizer shapes how much of that capacity is used in the induced weight-space updates. In a case study of GPT-2 adaptation with LoRA, we observe a strong rank-dependent optimizer effect. Despite using the same nominal rank, AdamW often produces per-step updates with concentrated singular spectra and low effective rank, whereas Muon uses a richer set of directions and benefits more consistently from increasing LoRA rank. These observations motivate ISO-LoRA, an optimizer that couples the LoRA factor updates through spectral descent on the induced tangent perturbation in weight space. ISO-LoRA promotes updates that distribute energy more evenly across singular directions, improving rank utilization while preserving compatibility with the LoRA parameterization. We complement this design with theoretical guarantees showing that ISO-LoRA can achieve higher effective rank than standard factor-wise optimizers through a one-step analysis under a stylized spiked-gradient model. We validate this design on language-model adaptation across 0.1B-7B-parameter models, where ISO-LoRA improves effective rank and downstream performance, with the strongest gains at moderate-to-large LoRA ranks. Our results highlight rank utilization as a key factor in LoRA optimization and suggest that optimizer design offers an important path toward stronger parameter-efficient adaptation.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-14 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Optimization and Control (math.OC); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2609.12123 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T03:05:44.017Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).