Metraipolis › archive › dc_636fdca3cddb
Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs — arXiv:2608.26650
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.26650
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T15:19:45.279Z
- SHA-256 of the archived text
38474a250c10af7ed2189e1c9f17e80aa1a9f379be58fc8b9a36467ac734f6a6- Stable URL of this page
https://metraipolis.com/documento/dc_636fdca3cddb
How to cite
Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs — arXiv:2608.26650 — arXiv (en), https://arxiv.org/abs/2608.26650 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_636fdca3cddb (Metraipolis archive dc_636fdca3cddb, sha256 38474a250c10af7ed2189e1c9f17e80aa1a9f379be58fc8b9a36467ac734f6a6)
BibTeX
@misc{dc636fdca3cddb,
title = {Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs — arXiv:2608.26650},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.26650},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_636fdca3cddb; sha256 38474a250c10af7ed2189e1c9f17e80aa1a9f379be58fc8b9a36467ac734f6a6},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26650. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Mixture-of-Experts (MoE) models route each token to a subset of expert networks, increasing capacity while keeping per-token computation sparse. In many deployed MoEs, the number of active experts is fixed across layers and tasks, although layer roles and expert redundancy vary with depth and demand varies with difficulty. Existing approaches address only part of this setting: layer-wise allocations are usually determined offline and reused for all tasks, while token-level methods vary expert activation using local routing signals without task-level context. We propose MetaNet, a support-set controller that predicts, for each layer, an expert-retention threshold and a bounded routing bias. The backbone, experts, and router remain frozen. On DeepSeek-MoE-16B-Chat, MetaNet provides a tunable accuracy-expert-activation trade-off. Relative to fixed k=6, a conservative setting activates 3.61 experts on average (40% fewer) and achieves comparable MMLU accuracy (0.489 vs. 0.474), whereas an aggressive setting activates 2.28 experts on average (62% fewer) with accuracy approximately 3.7 percentage points lower. The MMLU-trained controller also transfers to C-Eval without retraining, activating 2.90 experts on average (52% fewer than fixed k=6) at 0.386 accuracy. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.26650 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T15:15:18.065Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).