Metraipolis › archive › dc_ee40f5d63e07
Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers — arXiv:2609.22199
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.22199
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T07:06:12.886Z
- SHA-256 of the archived text
e7ddede22d78a8d2c41d4b741b7a3ee56ee49da2fa7b8c688b883f252b9cd24e- Stable URL of this page
https://metraipolis.com/documento/dc_ee40f5d63e07
How to cite
Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers — arXiv:2609.22199 — arXiv (en), https://arxiv.org/abs/2609.22199 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ee40f5d63e07 (Metraipolis archive dc_ee40f5d63e07, sha256 e7ddede22d78a8d2c41d4b741b7a3ee56ee49da2fa7b8c688b883f252b9cd24e)
BibTeX
@misc{dcee40f5d63e07,
title = {Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers — arXiv:2609.22199},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.22199},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ee40f5d63e07; sha256 e7ddede22d78a8d2c41d4b741b7a3ee56ee49da2fa7b8c688b883f252b9cd24e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.22199. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Transformer-based large language models often suffer from inter-layer parameter redundancy, where functional transformations are redundantly learned across network depths. We propose CS-MoE, a novel Transformer architecture featuring cross-layer expert sharing to address this inefficiency. Deviating from the widely used Mixture-of-Experts (MoE) architecture that terminates each Transformer block with layer-isolated experts, CS-MoE combines layer-independent experts with concurrent access to a centralized, globally shared expert pool. This \textit{Global Experts Sharing} mechanism enables elastic control over token-level parameter activation and computational consumption (FLOPs). Experiments demonstrate that CS-MoE achieves lower perplexity than equal-scale dense Transformers while activating only 55\% of parameters. Furthermore, its performance scales monotonically with an increased number of activated experts and approaches MoE counterparts that consume more FLOPs by expanding the shared pool with a fixed FLOPs budget. CS-MoE also establishes a flexible Pareto frontier between computational cost and model capacity, offering an efficient alternative for computation-constrained environments.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.22199 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:58:09.159Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).