Metraipolis › archive › dc_0bc47decd93d

From What Is Used to What Is Missing: Rethinking Dynamic Rank in LoRA — arXiv:2611.07349

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.07349
License (as archived): CC0-1.0
Archived: 2026-11-06T14:51:08.582Z
SHA-256 of the archived text: 1b131119ce908600dd51f619844c390d0817647a5687fc4ad43f47841f7e66e8
Stable URL of this page: https://metraipolis.com/documento/dc_0bc47decd93d

How to cite

From What Is Used to What Is Missing: Rethinking Dynamic Rank in LoRA — arXiv:2611.07349 — arXiv (en), https://arxiv.org/abs/2611.07349 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0bc47decd93d (Metraipolis archive dc_0bc47decd93d, sha256 1b131119ce908600dd51f619844c390d0817647a5687fc4ad43f47841f7e66e8)

BibTeX

@misc{dc0bc47decd93d,
  title = {From What Is Used to What Is Missing: Rethinking Dynamic Rank in LoRA — arXiv:2611.07349},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.07349},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0bc47decd93d; sha256 1b131119ce908600dd51f619844c390d0817647a5687fc4ad43f47841f7e66e8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07349. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Parameter-efficient fine-tuning with LoRA is governed by rank, which controls parameter budget and locally accessible task updates. Existing dynamic-rank methods mainly focus on capacity allocation, leaving unresolved whether the current adaptation space is sufficient for the task. We characterize the first-order information boundary of standard factor gradients, showing that their unobservable subspace coincides with the normal space of the low-rank manifold. Thus, factor gradients alone cannot distinguish under-optimization within the current space from task-relevant directions beyond its first-order reach. This motivates a geometric view of rank evolution, where tangent information optimizes existing degrees of freedom and the normal spectrum quantifies the value of new ones. Under a local quadratic model, we show that leading normal singular directions yield the locally optimal rank expansion, with spectral energy determining its gain. Building on this characterization, we propose GEAR, which turns the tangent--normal decomposition into rank evolution through tangent-space optimization, matrix-free normal-spectrum estimation for rank expansion, and task-induced geometric initialization. Across language and vision benchmarks, GEAR achieves the best average performance in all evaluated settings with substantially fewer trainable parameters. On LLaMA3-8B, it raises the average commonsense score from 86.61 to 88.45 while reducing trainable parameters from 42.0M to 14.4M.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Optimization and Control (math.OC) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.07349 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:21:42.657Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).