Metraipolis › archive › dc_d765364348a2

Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE — arXiv:2610.27598

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.27598
License (as archived)
CC0-1.0
Archived
2026-10-21T07:06:41.110Z
SHA-256 of the archived text
2a0e9124c9828f6bd7bcd419d313042e503c9dd1987404374c2285b6355104d7
Stable URL of this page
https://metraipolis.com/documento/dc_d765364348a2

How to cite

Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE — arXiv:2610.27598 — arXiv (en), https://arxiv.org/abs/2610.27598 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d765364348a2 (Metraipolis archive dc_d765364348a2, sha256 2a0e9124c9828f6bd7bcd419d313042e503c9dd1987404374c2285b6355104d7)
BibTeX
@misc{dcd765364348a2,
  title = {Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE — arXiv:2610.27598},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.27598},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d765364348a2; sha256 2a0e9124c9828f6bd7bcd419d313042e503c9dd1987404374c2285b6355104d7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.27598. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content. Compared to the hierarchically arranged \textbf{Local Expert} in existing mainstream models, we hypothesize that inter-layer shared \textbf{Meta Expert} hold greater potential. To this end, we conduct a multi-dimensional empirical comparison between Local and Meta Experts, and find that models incorporating Meta Experts exhibit lower parameter redundancy, together with lower cross-entropy loss and higher downstream task accuracy. With the activated quantity and the expert granularity held fixed, \textbf{our model matches the performance of Vanilla MoE using only 70--80\% of the experts}, substantially improving the model's scaling potential. Beyond removing inter-layer redundancy among experts and improving information storage efficiency, our theoretical analysis further shows that, compared with Local Experts, Meta Experts provide hidden states with more flexible residual pathways, thereby improving information retrieval efficiency.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.27598 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-21T07:05:43.425Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).