Metraipolis › archive › dc_81f55eb24737
MoSAR: Mixture of Semantic Attention Regimes for Learning Adaptive and Approximable Attention Geometries — arXiv:2609.31261
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.31261
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T01:39:47.619Z
- SHA-256 of the archived text
c04f2057ab945f8ce6e7318d2ef9dd918717c79f7d3199fa9a21e8adfe7f27b3- Stable URL of this page
https://metraipolis.com/documento/dc_81f55eb24737
How to cite
MoSAR: Mixture of Semantic Attention Regimes for Learning Adaptive and Approximable Attention Geometries — arXiv:2609.31261 — arXiv (en), https://arxiv.org/abs/2609.31261 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_81f55eb24737 (Metraipolis archive dc_81f55eb24737, sha256 c04f2057ab945f8ce6e7318d2ef9dd918717c79f7d3199fa9a21e8adfe7f27b3)
BibTeX
@misc{dc81f55eb24737,
title = {MoSAR: Mixture of Semantic Attention Regimes for Learning Adaptive and Approximable Attention Geometries — arXiv:2609.31261},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.31261},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_81f55eb24737; sha256 c04f2057ab945f8ce6e7318d2ef9dd918717c79f7d3199fa9a21e8adfe7f27b3},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.31261. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The quadratic complexity of dense self-attention remains a central bottleneck for long-context language modeling. Many efficient alternatives address this cost by deciding in advance where attention should be sparse or local. We argue that attention approximation should instead be approached as a geometric problem, with the relevant interaction geometry learned from data: natural-language dependencies are input-dependent and difficult to prescribe in advance, so the model should learn where positional relevance can decay and where broader interactions must be preserved. We introduce Mixture of Semantic Attention Regimes (MoSAR), which learns such an adaptive, controlled-decay geometry over query--key interactions. Input-conditioned query and key routers, applied after positional encoding, select mixtures over short, medium, and global regimes, inducing a continuous distance-dependent attention field rather than a fixed sparsity pattern. This geometry is learned during training and can subsequently be discretized through top-1 routing. In controlled pre-training experiments with matched 500M-parameter models, MoSAR learns a substantially lower-reach attention geometry without degrading language-modeling quality, improving perplexity over dense RoPE at the training context length. Under length extrapolation, MoSAR achieves the best perplexity among all evaluated variants, including strong baselines such as ALiBi. Moreover, the learned geometry remains stable under deterministic top-1 discretization, suggesting that it is not only adaptive, but also amenable to low-cost approximation at inference time. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-28 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.31261 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T01:12:14.561Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).