Metraipolis › archive › dc_bf61516cb00a

SMat-Attention: Structured Long-Context Sequence Modeling — arXiv:2609.36062

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.36062
License (as archived): CC0-1.0
Archived: 2026-10-07T16:22:13.221Z
SHA-256 of the archived text: 05ca3a407f60cfc39c7f4e16fa72fc6f268189e26cc789aecfaddbcc1a9080bc
Stable URL of this page: https://metraipolis.com/documento/dc_bf61516cb00a

How to cite

SMat-Attention: Structured Long-Context Sequence Modeling — arXiv:2609.36062 — arXiv (en), https://arxiv.org/abs/2609.36062 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf61516cb00a (Metraipolis archive dc_bf61516cb00a, sha256 05ca3a407f60cfc39c7f4e16fa72fc6f268189e26cc789aecfaddbcc1a9080bc)

BibTeX

@misc{dcbf61516cb00a,
  title = {SMat-Attention: Structured Long-Context Sequence Modeling — arXiv:2609.36062},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.36062},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf61516cb00a; sha256 05ca3a407f60cfc39c7f4e16fa72fc6f268189e26cc789aecfaddbcc1a9080bc},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36062. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Long-context sequence models face a fundamental tradeoff: softmax attention uses flexible token-level interactions at quadratic cost, whereas linear attention obtains linear-time training and constant-time decoding by compressing history into a fixed-size state. In this work, we ask whether we can connect these regimes through a tunable notion of structure. To this end, we introduce Structured Matrix Attention (SMat-Attention) via a family of causal masks with structured long-range routing whose row supports have VC-dimension $d$. In our construction, $d=1$ recovers the standard causal mask, and increasing $d$ permits richer subset-routing patterns. We give chunkwise forward and backward algorithms to enable hardware-efficiency. For sequences of length $T$, the hard-routing construction takes $O(T^{2-3/d}+T)$ work, despite the mask being dense, for our prescribed family. In fixed-horizon streaming, decoding after the distant prefix takes constant time per token using $O(T^{1-1/d})$ cached states. SMat-Attention therefore makes VC-dimension an explicit knob governing access-pattern complexity, prefill cost, and decoding memory. Empirically, subset-routing and rule-assisted multi-key retrieval experiments illustrate the masks' routing expressiveness. Extensions to Mamba-2 and Gated DeltaNet using learned routing with top-$k$ query reads retain subquadratic prefill, improve recall accuracy over the backbones in several settings, and achieve comparable small-scale language-modeling performance.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.36062 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:53.460Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).