Metraipolis › archive › dc_f4529d1efc87
MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuning — arXiv:2608.15311
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.15311
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T17:13:49.973Z
- SHA-256 of the archived text
dbfdca72cf5b33fedfe6bc20c4109cf97dc8dcdd2444d01c576e8a4f600f6a51- Stable URL of this page
https://metraipolis.com/documento/dc_f4529d1efc87
How to cite
MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuning — arXiv:2608.15311 — arXiv (en), https://arxiv.org/abs/2608.15311 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f4529d1efc87 (Metraipolis archive dc_f4529d1efc87, sha256 dbfdca72cf5b33fedfe6bc20c4109cf97dc8dcdd2444d01c576e8a4f600f6a51)
BibTeX
@misc{dcf4529d1efc87,
title = {MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuning — arXiv:2608.15311},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.15311},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f4529d1efc87; sha256 dbfdca72cf5b33fedfe6bc20c4109cf97dc8dcdd2444d01c576e8a4f600f6a51},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.15311. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Federated instruction fine-tuning enables Large Language Models (LLMs) to adapt to decentralized, privacy-sensitive data without requiring data sharing. Recent Mixture-of-Experts (MoE) LLMs are particularly attractive for federated learning because their sparse activation reduces computation and communication while scaling model capacity. However, existing federated MoE methods primarily focus on parameter aggregation and personalization, overlooking the routing behavior of MoE models as a source of information for client collaboration. Under heterogeneous instruction distributions, indiscriminate aggregation can lead to negative transfer, highlighting the need to identify which clients should collaborate during federated optimization. We propose ClientMorpher, a routing-aware, personalized federated instruction fine-tuning framework that leverages routing signatures from pretrained MoE models to organize client collaboration prior to aggregation. We investigate two complementary clustering strategies: ClientMorpher-C, which directly clusters clients using expert activation profiles, and ClientMorpher-E, which first clusters experts based on their cross-client usage signatures and then derives client collaboration groups. We evaluate ClientMorpher for federated instruction fine-tuning on the Databricks Dolly-15K dataset, using pathological and Dirichlet-based heterogeneous client distributions across multiple instruction-following tasks. Experimental results show that routing-aware collaboration consistently improves personalized performance compared to conventional federated averaging and local training, while maintaining the same communication cost. Furthermore, our study shows that client-centric and expert-centric clustering provides an effective and scalable approach for personalized federated instruction fine-tuning of sparse MoE LLMs. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.15311 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T16:29:13.789Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).