Metraipolis › archive › dc_a6ee5d40e607
Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion — arXiv:2610.19232
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.19232
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T14:15:18.786Z
- SHA-256 of the archived text
3b40b8830bbfc83cb4bf1cfc12a03c2ad2144ae6099af1d05b1f34a2038f4ee9- Stable URL of this page
https://metraipolis.com/documento/dc_a6ee5d40e607
How to cite
Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion — arXiv:2610.19232 — arXiv (en), https://arxiv.org/abs/2610.19232 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a6ee5d40e607 (Metraipolis archive dc_a6ee5d40e607, sha256 3b40b8830bbfc83cb4bf1cfc12a03c2ad2144ae6099af1d05b1f34a2038f4ee9)
BibTeX
@misc{dca6ee5d40e607,
title = {Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion — arXiv:2610.19232},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.19232},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a6ee5d40e607; sha256 3b40b8830bbfc83cb4bf1cfc12a03c2ad2144ae6099af1d05b1f34a2038f4ee9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19232. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models impose substantial memory demands and arithmetic costs during inference. Dense-to-mixture-of-experts (MoE) conversion is a model optimization technique that aims to reduce these arithmetic costs. Dense-to-MoE conversion partitions pretrained feed-forward network (FFN) neurons into experts and selects a subset for each token. Leading methods fall into two categories: analytic methods produce a converted model quickly using heuristics, while learning methods produce a model much more slowly by optimizing via gradients. Both approaches, however, fix the same active expert count, or "utilization," at every layer. We introduce Spectral-MoE, which learns to allocate a utilization budget heterogeneously across layers. Its name draws a metaphor between separating a dense FFN into experts and separating a light beam into its spectrum. Spectral-MoE consists of two distinct approaches: a static approach that uses a single per-layer scalar which is shared across tokens, and a dynamic approach which emits a per-token schedule. We demonstrate that both allocators improve performance over uniform on both the analytic-based and learning-based SOTA dense-to-MoE methods. At 25% FFN utilization, dynamic allocation reduces WikiText-2 perplexity relative to uniform by 1.04, 2.96, and 2.71 points on the learned base for Llama-2-7B, Llama-3-8B, and Qwen2.5-7B, respectively. The corresponding reductions on the analytic base are 2.45, 6.26, and 2.17 points. On the learned base, dynamic allocation improves six-task mean accuracy over uniform by an average of 1.5, 4.2, and 3.6 percentage points across the three models at FFN utilizations of 12.5%, 25%, and 50%, respectively. These results establish per-layer expert utilization as a novel optimization axis in dense-to-MoE conversion. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.19232 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T13:53:35.546Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).