Metraipolis › archive › dc_bcb0fc121c2f

Rethinking LoRA Initialization for Robust Asymmetric Learning Rates — arXiv:2610.32142

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.32142
License (as archived): CC0-1.0
Archived: 2026-10-26T19:51:43.089Z
SHA-256 of the archived text: d8bd324b3cbecf6defd910630bd9e143a261ecda13a876ed34e2271664ca536e
Stable URL of this page: https://metraipolis.com/documento/dc_bcb0fc121c2f

How to cite

Rethinking LoRA Initialization for Robust Asymmetric Learning Rates — arXiv:2610.32142 — arXiv (en), https://arxiv.org/abs/2610.32142 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bcb0fc121c2f (Metraipolis archive dc_bcb0fc121c2f, sha256 d8bd324b3cbecf6defd910630bd9e143a261ecda13a876ed34e2271664ca536e)

BibTeX

@misc{dcbcb0fc121c2f,
  title = {Rethinking LoRA Initialization for Robust Asymmetric Learning Rates — arXiv:2610.32142},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.32142},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bcb0fc121c2f; sha256 d8bd324b3cbecf6defd910630bd9e143a261ecda13a876ed34e2271664ca536e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.32142. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Low-Rank Adaptation (LoRA) is often highly sensitive to learning-rate (LR) choices. Recent empirical studies suggest that, once LRs are carefully tuned, vanilla LoRA can remain competitive with many proposed LoRA variants. This shifts the central practical question from designing ever more variants to reducing the cost and brittleness of LR selection. LoRA+ addresses this issue with asymmetric LRs, using a larger LR for the up-projection matrix $B$ than for the down-projection matrix $A$. However, the ratio $\eta_B / \eta_A$ itself becomes an additional hyperparameter: large ratios can improve adaptation, but under the standard LoRA initialization they can also cause unstable training and ratio-dependent performance collapse. We identify the source of this brittleness through a signal--noise decomposition of LoRA dynamics. Under standard \initA, the update contains an initialization-induced noise term whose magnitude can be affected by the LR for $B$, $\eta_B$, the initialization scale $\beta$, and the adapter rank $r$. In practical LLM fine-tuning, this non-vanishing noise can limit the performance gains expected from LoRA+ style asymmetric LRs. To suppress this effect, we propose \initAA, a simple initialization that scales the variance of $A$ as $\sigma_A^2 \propto n^{-2}$ while keeping $B_0=0$. Under matched rank and training time for both initializations, this reduces the initialization-noise energy by an additional factor of $1/n$, stabilizing asymmetric LoRA training while preserving zero initial adapter output. Empirically, \initAA substantially widens the stable LR region across initialization scales, adapter ranks, and LoRA update multipliers. Across tasks and models, \initAA reduces best-ratio drift, prevents the high-ratio collapse observed with standard \initA, and makes the width-based guideline $\eta_B \approx n \eta_A$ a useful starting point for finite pretrained models.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.32142 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-26T19:45:33.389Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).