Metraipolis › archive › dc_fd98c19ff461

How Finite-Rank Bottlenecks Shape the Low-Rank Adaptation Landscape — arXiv:2610.23343

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.23343
License (as archived): CC0-1.0
Archived: 2026-10-19T08:48:44.948Z
SHA-256 of the archived text: 0f970756cd271686d64af8efe37b13e2355eccf83d97fddffc8f0c31e2626472
Stable URL of this page: https://metraipolis.com/documento/dc_fd98c19ff461

How to cite

How Finite-Rank Bottlenecks Shape the Low-Rank Adaptation Landscape — arXiv:2610.23343 — arXiv (en), https://arxiv.org/abs/2610.23343 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fd98c19ff461 (Metraipolis archive dc_fd98c19ff461, sha256 0f970756cd271686d64af8efe37b13e2355eccf83d97fddffc8f0c31e2626472)

BibTeX

@misc{dcfd98c19ff461,
  title = {How Finite-Rank Bottlenecks Shape the Low-Rank Adaptation Landscape — arXiv:2610.23343},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.23343},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fd98c19ff461; sha256 0f970756cd271686d64af8efe37b13e2355eccf83d97fddffc8f0c31e2626472},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.23343. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Low-rank adaptation (LoRA) has become the standard method for parameter-efficient fine-tuning of large pretrained models, yet theoretical explanations for why low-rank updates suffice remain incomplete. Most existing analyses rely on the Neural Tangent Kernel (NTK) regime, which linearizes the network around the pretrained weights. In this work, we take a different approach that requires no linearization: our key observation is that, once the backbone is frozen, the loss sees each adapted weight matrix only through a finite-rank linear map induced by the frozen activations. Trace-norm regularization then forces the optimizer to pick a low-rank matrix among all matrices that produce the same observed output. This yields global minimizers whose ranks are controlled by the rank of this map, which we call the \emph{bottleneck rank}, rather than the sample size. For multi-head attention, we obtain tighter rank upper bounds by exploiting two additional invariances: row-wise softmax is insensitive to additive row constants in the score matrix, and value updates are observable only after attention weighting and frozen output projection. For the factorized LoRA objective with weight decay, we prove a complementary result on the nonconvex side: after an arbitrarily small generic positive semidefinite perturbation, every first-order stationary point is rank-deficient once the LoRA rank exceeds a threshold determined by the bottleneck rank. Finally, we show that global minimizers of the perturbed objective are near-optimal for the original trace-norm problem, and we provide uniform generalization bounds governed by the spectral structure of the frozen features.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.23343 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:42:40.276Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).