Metraipolis › archive › dc_aefd952d6820
DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition — arXiv:2610.11317
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.11317
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T05:47:36.167Z
- SHA-256 of the archived text
76c1a4bcb24b34dfb9b27087fab4752bfa9e33fe8ffbb8b38d0995bc7093f446- Stable URL of this page
https://metraipolis.com/documento/dc_aefd952d6820
How to cite
DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition — arXiv:2610.11317 — arXiv (en), https://arxiv.org/abs/2610.11317 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aefd952d6820 (Metraipolis archive dc_aefd952d6820, sha256 76c1a4bcb24b34dfb9b27087fab4752bfa9e33fe8ffbb8b38d0995bc7093f446)
BibTeX
@misc{dcaefd952d6820,
title = {DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition — arXiv:2610.11317},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.11317},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aefd952d6820; sha256 76c1a4bcb24b34dfb9b27087fab4752bfa9e33fe8ffbb8b38d0995bc7093f446},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11317. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, with recent work demonstrating the benefits of fine-grained expert designs. Training such models from scratch is expensive, and sparse upcycling from pre-trained dense models is an attractive alternative. However, we identify a structural pathology of fine-grained upcycling: when fine-grained experts are derived from a single source model, naive routing collapses and downstream accuracy drops to near-random (e.g., on Qwen3-1.7B, Drop-Upcycling-fine-grained reaches only 23.2% average accuracy across 15 benchmarks, essentially matching from-scratch training at 22.2%, while the same method's coarse-grained variant reaches 50.2%). We propose DivMoE, the first framework achieving fine-grained MoE Upcycling with structurally-balanced routing. DivMoE introduces domain-specialized fine-grained expert initialization, deriving experts from dense models that have undergone domain-adaptive continual pre-training, and diversity-constrained routing, a hard structural constraint guaranteeing that each token activates experts from distinct domain groups. Across two base models and 15 benchmarks, DivMoE consistently outperforms six upcycling baselines (55.6% vs. 51.6% for the strongest baseline on Qwen3-1.7B) and strictly improves over the dense base model on every benchmark after Stage 2 continual pre-training -- closing the regression gap that has plagued prior fine-grained upcycling. After supervised fine-tuning on a public reasoning mixture, our 12B-parameter DivMoE model matches Moonlight-MoE (16B) at 64.5% average accuracy while outperforming a controlled NVIDIA-Upcycling baseline by 6.1 percentage points. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.11317 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T05:47:00.612Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).