Metraipolis › archive › dc_829c71fd38fd

Attention-Aware Routing: Coupling Routing and Attention in MoEs — arXiv:2609.20974

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.20974
License (as archived)
CC0-1.0
Archived
2026-10-10T01:01:34.350Z
SHA-256 of the archived text
d7c345af1e62038af55703643009bd852c6388405904f125b202f13798fc0f11
Stable URL of this page
https://metraipolis.com/documento/dc_829c71fd38fd

How to cite

Attention-Aware Routing: Coupling Routing and Attention in MoEs — arXiv:2609.20974 — arXiv (en), https://arxiv.org/abs/2609.20974 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_829c71fd38fd (Metraipolis archive dc_829c71fd38fd, sha256 d7c345af1e62038af55703643009bd852c6388405904f125b202f13798fc0f11)
BibTeX
@misc{dc829c71fd38fd,
  title = {Attention-Aware Routing: Coupling Routing and Attention in MoEs — arXiv:2609.20974},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.20974},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_829c71fd38fd; sha256 d7c345af1e62038af55703643009bd852c6388405904f125b202f13798fc0f11},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.20974. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In Mixture-of-Experts language models, the router typically selects and weights experts based on the token's hidden state, utilizing limited contextual information. We propose Attention-Aware Routing (AAR), which augments the router with temporal and spectral features extracted from a sliding window of attention weights that represent a summary of the model's contextual state, disentangled from the hidden state. Keeping the base transformer entirely frozen, we train only the routing parameters, isolating routing as the sole variable. AAR improves GSM8K by +3.37 pp over a routing-only SFT baseline on OLMoE. Beyond performance, we show that routing and attention form a coupled circuit: routing changes at layer l propagate through the residual stream to amplify attention sinks at layer l+1, reshaping attention without any direct update to the attention mechanism itself. Further, AAR reduces long diverging generation, with incorrect answers getting shorter, while correct answers remain unchanged in length. Finally, AAR is strongly depth-sensitive: applying it indiscriminately across layers can degrade factual retrieval, whereas mathematical reasoning gains persist when it is introduced deeper in the network. This sensitivity exposes a retrieval--reasoning tension across depth and makes layer-selective AAR a controlled probe of the routing-relevant information carried by attention at different layers.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.20974 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T00:23:42.423Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).