Metraipolis › archive › dc_fb089eca8647

From Bilinear to Linear: Differentially Private Federated LoRA via Low-Dimensional Parameterization — arXiv:2609.26091

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.26091
License (as archived)
CC0-1.0
Archived
2026-10-08T14:42:03.895Z
SHA-256 of the archived text
ac8dcc3ac7d71d16f85dce6350fcb7078e6343553cad21f972f51ff61ea56050
Stable URL of this page
https://metraipolis.com/documento/dc_fb089eca8647

How to cite

From Bilinear to Linear: Differentially Private Federated LoRA via Low-Dimensional Parameterization — arXiv:2609.26091 — arXiv (en), https://arxiv.org/abs/2609.26091 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fb089eca8647 (Metraipolis archive dc_fb089eca8647, sha256 ac8dcc3ac7d71d16f85dce6350fcb7078e6343553cad21f972f51ff61ea56050)
BibTeX
@misc{dcfb089eca8647,
  title = {From Bilinear to Linear: Differentially Private Federated LoRA via Low-Dimensional Parameterization — arXiv:2609.26091},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.26091},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fb089eca8647; sha256 ac8dcc3ac7d71d16f85dce6350fcb7078e6343553cad21f972f51ff61ea56050},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.26091. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Federated Low-Rank Adaptation (LoRA) provides an efficient solution for finetuning large language models across distributed and privacy-sensitive data. However, despite avoiding raw data sharing, federated LoRA remains vulnerable to privacy leakage through transmitted model updates. Differential privacy (DP) mitigates such leakage, but integrating DP into federated LoRA introduces two fundamental challenges: aggregation mismatch from independently averaging low-rank factors, and quadratic noise amplification when noise is injected into both factors. To address these challenges, we propose FedHSIP, a differentially private federated LoRA framework based on a unified low-dimensional parameterization. FedHSIP reformulates all LoRA parameters into a shared low-dimensional trainable vector, enabling clients to optimize and communicate only low-dimensional updates. This reformulation transforms federated LoRA from a bilinear factor aggregation problem into a unified linear parameter space, thereby eliminating aggregation mismatch and preventing the quadratic amplification of DP noise. To further handle non-IID data, we introduce a heterogeneity- and sensitivity-aware isometric projection, constructed from warm-up statistics, which groups coordinates with compatible cross-client update patterns while balancing sensitivity, update energy, and heterogeneity across the low-dimensional space. Extensive experiments on natural language understanding and generation benchmarks show that FedHSIP consistently outperforms existing federated LoRA methods under both private and non-private settings, achieving up to 3-4% improvements under differential privacy while reducing communication cost by over 80% and maintaining robustness under heterogeneous data distributions.

Tema degli esploratori: fiducia digitale e crittografia (retro-catalogo arXiv: annunci del 2026-09-23 in cs.CR).
Categorie arXiv: Cryptography and Security (cs.CR).
Pagina del lavoro: https://arxiv.org/abs/2609.26091 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T13:55:11.739Z dalla pagina https://arxiv.org/catchup/cs.CR/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).