Metraipolis › archive › dc_bbb073b48616

Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts — arXiv:2609.01100

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.01100
License (as archived)
CC0-1.0
Archived
2026-10-17T06:41:28.508Z
SHA-256 of the archived text
1ebe4db2a8e747466bebdc81fc492b81bd2dd42eefd567292d60d6e8b56a31e4
Stable URL of this page
https://metraipolis.com/documento/dc_bbb073b48616

How to cite

Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts — arXiv:2609.01100 — arXiv (en), https://arxiv.org/abs/2609.01100 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bbb073b48616 (Metraipolis archive dc_bbb073b48616, sha256 1ebe4db2a8e747466bebdc81fc492b81bd2dd42eefd567292d60d6e8b56a31e4)
BibTeX
@misc{dcbbb073b48616,
  title = {Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts — arXiv:2609.01100},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.01100},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bbb073b48616; sha256 1ebe4db2a8e747466bebdc81fc492b81bd2dd42eefd567292d60d6e8b56a31e4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01100. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In current Mixture-of-Experts architectures, routing is performed based on representations dominated by structure shared across all tokens, limiting expert specialization. We show that contrasting each token against an Exponential Moving Average of the layer's hidden states, rather than routing on absolute magnitude, concentrates the routing signal onto a low-dimensional, highly separable subspace. Building on this, we propose the Contrastive Routing Mechanism (CoRM), which scores each expert by the gap between its affinity for the incoming token and its affinity for this shared reference state, interpreted through a distinct per-expert projection. The resulting experts have routing boundaries that align with linguistic structure significantly more than the Top-k baseline. Our experiments show that CoRM improves average zero-shot accuracy by +0.67 to +1.69 points (Top-1) and +1.38 to +1.77 points (Top-2) over standard Top-k MoE baselines on nine zero-shot reasoning benchmarks, at the minimal cost of 2.9% added parameters and 2.6% added FLOPs per token.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.01100 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:33:38.842Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).