Metraipolis › archive › dc_4657f2faf64b
MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling — arXiv:2610.07809
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.07809
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T11:41:23.496Z
- SHA-256 of the archived text
305ea81249ba224a17ee70d2b680a066bb403fd7d26237751e5509957a951cf4- Stable URL of this page
https://metraipolis.com/documento/dc_4657f2faf64b
How to cite
MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling — arXiv:2610.07809 — arXiv (en), https://arxiv.org/abs/2610.07809 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4657f2faf64b (Metraipolis archive dc_4657f2faf64b, sha256 305ea81249ba224a17ee70d2b680a066bb403fd7d26237751e5509957a951cf4)
BibTeX
@misc{dc4657f2faf64b,
title = {MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling — arXiv:2610.07809},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.07809},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4657f2faf64b; sha256 305ea81249ba224a17ee70d2b680a066bb403fd7d26237751e5509957a951cf4},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07809. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Sparsely activated Mixture-of-Experts (MoE) models increase model capacity without a proportional increase in per-token computation. Dense-to-MoE upcycling reuses pretrained dense models to construct such systems, commonly by copying feed-forward networks (FFNs) into independently trained experts. We introduce MASKerade, a dense-to-MoE training method that instead learns experts as sparse subnetworks of a frozen pretrained FFN. Each expert is defined by a learned binary mask, and a token-level router selects which masked FFNs to execute and combine. The router and mask scores are optimized jointly, while the underlying FFN weight values remain unchanged. This formulation supports neuron-structured, semi-structured, and unstructured experts within the same routing architecture. Our main configuration uses four 2:4 experts with top-2 routing, where two half-dense expert passes have the nominal FFN arithmetic of one dense pass, without requiring independent expert weight matrices. On five vision-language benchmarks with Qwen and Gemma backbones, this configuration achieves the highest performance among the compared baselines. Comparisons across mask granularities, routing interventions, and compute-matched controls distinguish the effects of learned connectivity from expert activation count. These results establish mask learning over frozen weights as a practical alternative for constructing token-routed MoE experts. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.07809 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T11:19:24.787Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).