Metraipolis › archive › dc_c19a46be4aef

One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning — arXiv:2609.05885

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.05885
License (as archived): CC0-1.0
Archived: 2026-10-16T13:56:25.103Z
SHA-256 of the archived text: 76d2cc645fefd170dbcc8623607cb215c8539f71fe3a25b1e1973a935ba86495
Stable URL of this page: https://metraipolis.com/documento/dc_c19a46be4aef

How to cite

One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning — arXiv:2609.05885 — arXiv (en), https://arxiv.org/abs/2609.05885 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c19a46be4aef (Metraipolis archive dc_c19a46be4aef, sha256 76d2cc645fefd170dbcc8623607cb215c8539f71fe3a25b1e1973a935ba86495)

BibTeX

@misc{dcc19a46be4aef,
  title = {One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning — arXiv:2609.05885},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.05885},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c19a46be4aef; sha256 76d2cc645fefd170dbcc8623607cb215c8539f71fe3a25b1e1973a935ba86495},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.05885. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Low-rank adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of large language models. Most LoRA variants follow a uniform-LR convention, applying a single global learning rate across every rank-one component of every adapter. We show that this convention overlooks substantial within-module heterogeneity, where the rank-one components of a LoRA adapter update at highly uneven rates and low-velocity modules converge to concentrated singular spectra that underutilize the nominal rank budget. To address this, we propose an adaptive anisotropic learning-rate model that assigns each rank-one component its own effective learning rate, computed online from training-time signals and mean-normalized per module to preserve the global LR budget. AnLR-LoRA instantiates this model with two signals available during AdamW optimization, namely function-space velocity and Adam SNR, as a lightweight scheme with no extra trainable parameters. Across commonsense reasoning, natural language generation and visual instruction-tuning benchmarks, AnLR-LoRA consistently improves over LoRA while encouraging broader use of rank capacity, with gains that remain robust across a wide range of global learning rates and transfer cleanly to other LoRA variants.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.05885 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T13:39:42.820Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).