Metraipolis › archive › dc_aa60930292e4

A Comparative Analysis of Attention versus State-Space Models for In-Context Learning — arXiv:2609.32341

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.32341
License (as archived)
CC0-1.0
Archived
2026-10-07T22:36:31.149Z
SHA-256 of the archived text
df2f02b38059f586e565b6afeedafc4dca6dc22d451d2100dfc1c23d28808229
Stable URL of this page
https://metraipolis.com/documento/dc_aa60930292e4

How to cite

A Comparative Analysis of Attention versus State-Space Models for In-Context Learning — arXiv:2609.32341 — arXiv (en), https://arxiv.org/abs/2609.32341 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aa60930292e4 (Metraipolis archive dc_aa60930292e4, sha256 df2f02b38059f586e565b6afeedafc4dca6dc22d451d2100dfc1c23d28808229)
BibTeX
@misc{dcaa60930292e4,
  title = {A Comparative Analysis of Attention versus State-Space Models for In-Context Learning — arXiv:2609.32341},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32341},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aa60930292e4; sha256 df2f02b38059f586e565b6afeedafc4dca6dc22d451d2100dfc1c23d28808229},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32341. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Transformers and state-space models (SSMs) are two prominent sequential learning architectures, yet their comparison remains largely empirical and existing theoretical analyses are typically task-specific or architecturally restricted. In this paper, we develop belief geometry, a unified analytical framework for comparing the representational capabilities of broad classes of attention and SSMs. Starting from a generalized formulation of in-context linear regression and using cumulative Bayes regret as our measure, we abstract three capabilities required by many sequential learning problems in our belief geometry: evidence assembly, belief maintenance, and addressing. We then study three cases of our formulation that isolate these capabilities and yield sharp architectural lessons: For belief maintenance, SSMs attain the optimal regret over stationary aggregation kernels; for positional assembly, SSMs have a memory advantage; and for content addressing, softmax attention has an exponential width advantage over sigmoid-selective SSMs. Experiments with LLaMA-type Transformers and Mamba-2 show that these architectural insights extend beyond our analytically tractable classes and linear-regression testbed.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.32341 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:27.663Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).