Metraipolis › archive › dc_511bf2d316ae

Switching Linear Attention — arXiv:2609.39034

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.39034
License (as archived): CC0-1.0
Archived: 2026-10-07T12:44:47.649Z
SHA-256 of the archived text: 91b610b175d228fb8c1e9051b534142d920141c02556ff0c1fadcedbe64d06bc
Stable URL of this page: https://metraipolis.com/documento/dc_511bf2d316ae

How to cite

Switching Linear Attention — arXiv:2609.39034 — arXiv (en), https://arxiv.org/abs/2609.39034 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_511bf2d316ae (Metraipolis archive dc_511bf2d316ae, sha256 91b610b175d228fb8c1e9051b534142d920141c02556ff0c1fadcedbe64d06bc)

BibTeX

@misc{dc511bf2d316ae,
  title = {Switching Linear Attention — arXiv:2609.39034},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.39034},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_511bf2d316ae; sha256 91b610b175d228fb8c1e9051b534142d920141c02556ff0c1fadcedbe64d06bc},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39034. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Designing expressive sequence layers with efficient inference remains a central challenge in modern machine learning. Standard softmax attention achieves excellent sequence modeling performance through rich nonlinear token interactions, but it requires a key-value cache that grows linearly with sequence length, limiting its scalability. Linear attention enables efficient recurrent computation with a constant memory footprint, yet its reduced expressivity often yields inferior modeling performance. We introduce Switching Linear Attention (SwiLA), a novel sequence layer that bridges this gap by enhancing representational capacity while retaining the fixed-size recurrent state of linear attention. We derive the SwiLA recurrence from the test-time regression framework, casting the state update rule as online expectation-maximization in a mixture of linear regressions model. At test time, each output dimension dynamically selects among multiple linear attention components based on the input. Across associative recall, in-context language learning, and language modeling benchmarks, SwiLA shows strong performance and narrows the gap to softmax attention, even surpassing it in several settings.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.39034 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:14.002Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).