Metraipolis › archive › dc_bbe6ebd4df05

Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach — arXiv:2608.09742

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.09742
License (as archived)
CC0-1.0
Archived
2026-10-19T08:52:52.452Z
SHA-256 of the archived text
4e58b8664ccd395b0ad4329fe38ac9605b4add83757b19ebd669c8a90b90e8ed
Stable URL of this page
https://metraipolis.com/documento/dc_bbe6ebd4df05

How to cite

Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach — arXiv:2608.09742 — arXiv (en), https://arxiv.org/abs/2608.09742 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bbe6ebd4df05 (Metraipolis archive dc_bbe6ebd4df05, sha256 4e58b8664ccd395b0ad4329fe38ac9605b4add83757b19ebd669c8a90b90e8ed)
BibTeX
@misc{dcbbe6ebd4df05,
  title = {Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach — arXiv:2608.09742},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.09742},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bbe6ebd4df05; sha256 4e58b8664ccd395b0ad4329fe38ac9605b4add83757b19ebd669c8a90b90e8ed},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.09742. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Low-rank adaptation (LoRA) represents large language model (LLM) updates with two compact matrix factors, i.e., $A$ and $B$, providing an efficient way to fine-tune large models in federated learning paradigm. Inspired by the asymmetric roles of the LoRA factors, we study whether $A$ should be shared across clients while $B$ remains client-specific (Share-A/Local-B), or whether $B$ should instead be shared while $A$ remains client-specific (Share-B/Local-A). With a least-squares surrogate, we reveal that Share-A/Local-B requires the client-specific LoRA update matrices to use a common rank-$r$ input-side space, whereas Share-B/Local-A requires a common rank-$r$ output-side space. The two strategies therefore incur different projection residuals, indicating that the preferred strategy is the one with the smaller aggregate residual across clients. With this insight, we propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance. To enable adaptive factor selection before training, we design a Rank-Aware Shared-Subspace Sufficiency (RSS) metric, which effectively assesses whether a shared rank-$r$ input subspace is sufficient for the local data distributions using representations extracted from a frozen LLM backbone. Experiments across different tasks, data distributions, LoRA ranks, and participation settings confirm the effectiveness of RSS and the superior performance of FedAS-LoRA.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-11 in cs.DC).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2608.09742 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:50:23.791Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-08-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).