Metraipolis › archive › dc_517fea036fbd
HERO-MoE: Historical Expert Routing with Scale-Preserving Fusion — arXiv:2609.32581
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.32581
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:50:13.237Z
- SHA-256 of the archived text
b2907bdea07b1fceb52a2142c5d815a0a6c53400a6998edb11e974a0d4e64e49- Stable URL of this page
https://metraipolis.com/documento/dc_517fea036fbd
How to cite
HERO-MoE: Historical Expert Routing with Scale-Preserving Fusion — arXiv:2609.32581 — arXiv (en), https://arxiv.org/abs/2609.32581 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_517fea036fbd (Metraipolis archive dc_517fea036fbd, sha256 b2907bdea07b1fceb52a2142c5d815a0a6c53400a6998edb11e974a0d4e64e49)
BibTeX
@misc{dc517fea036fbd,
title = {HERO-MoE: Historical Expert Routing with Scale-Preserving Fusion — arXiv:2609.32581},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.32581},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_517fea036fbd; sha256 b2907bdea07b1fceb52a2142c5d815a0a6c53400a6998edb11e974a0d4e64e49},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32581. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Mixture-of-Experts (MoE) architectures have become a standard way to scale model capacity while keeping computation sparse, yet routing remains a key determinant of MoE quality and training behavior. Prior empirical studies suggest that MoE routing reflects input semantics and upstream computation across depth, but standard routers do not explicitly use the routing distributions produced by preceding layers. We propose HERO-MoE, Historical Expert ROuting with Scale-Preserving Fusion, a routing framework that injects historical routing priors into MoE routers by reusing detached, dense routing distributions collected from preceding MoE layers. The key idea is simple: HERO-MoE preserves the original token-conditioned routing branch and adds a residual historical routing contribution before the standard softmax and top-$k$ dispatch. To stabilize this historical signal, HERO-MoE introduces a scale-preserving fusion mechanism that matches the magnitude of historical routing memory to the current hidden representation and accounts for the number of visible historical layers, without introducing an auxiliary routing loss or a fusion-specific tuning parameter. By reusing routing distributions already computed by preceding MoE layers, HERO-MoE improves training-loss reduction with modest end-to-end overhead. The resulting router remains compatible with standard sparse dispatch, including top-$k$ and group-limited routing, and can be inserted into existing MoE backbones with minimal architectural changes. Experiments on an approximately 8B-parameter MoE model with 0.5B active parameters, trained from scratch on 100B tokens, show that HERO-MoE reduces the final loss from 1.6393 to 1.6184, while peak memory and FLOPs increase by only 0.44\% and 0.64\%, respectively. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.32581 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T21:27:08.449Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).