Metraipolis › archive › dc_2e2bf4ef0563

FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon — arXiv:2609.06073

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.06073
License (as archived): CC0-1.0
Archived: 2026-10-16T13:56:32.749Z
SHA-256 of the archived text: d7d0cc2f61a03279b08762780360cf7f0ed0de4b46b7cdc105a6dd64a2626c53
Stable URL of this page: https://metraipolis.com/documento/dc_2e2bf4ef0563

How to cite

FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon — arXiv:2609.06073 — arXiv (en), https://arxiv.org/abs/2609.06073 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2e2bf4ef0563 (Metraipolis archive dc_2e2bf4ef0563, sha256 d7d0cc2f61a03279b08762780360cf7f0ed0de4b46b7cdc105a6dd64a2626c53)

BibTeX

@misc{dc2e2bf4ef0563,
  title = {FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon — arXiv:2609.06073},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.06073},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2e2bf4ef0563; sha256 d7d0cc2f61a03279b08762780360cf7f0ed0de4b46b7cdc105a6dd64a2626c53},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.06073. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Federated fine-tuning adapts large language models (LLMs) to decentralized client data, but its scalability in cross-device training is often limited by the high communication cost. Muon is an optimizer that improves optimization performance by orthogonalizing momentum for matrix-valued parameters. Existing federated Muon methods demonstrate the benefit of matrix-aware optimization in federated learning, but still require transmitting full layer-size updates and optimizer state. A natural way to reduce communication is to directly apply Muon to LoRA factors, but this changes the optimized object and weakens Muon's matrix-aware update geometry. We propose FedSubMuon, a communication-efficient federated Muon fine-tuning method that optimizes compact coefficient matrices within shared structured subspaces. This design keeps Muon on a single matrix-valued trainable object, while reducing the client upload to compact coefficient matrices. We further introduce FedSubMuon-GT, an accuracy-oriented extension that uses projected gradients to adapt tracked subspace bases toward task-relevant gradient directions. Experiments on instruction tuning and mathematical reasoning show that FedSubMuon-GT achieves the best overall accuracy on four of five dataset-model pairs, while FedSubMuon performs best under all matched communication budgets. On Dolly-15K, the closest communication baseline requires 5.5 times and 1.4 times more total communication on Llama-1B and Qwen-4B, respectively.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.06073 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T13:39:42.839Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).