Metraipolis › archive › dc_46cd7378d37d

SPARE: Fast Linear Attention Decoding with Compact Recurrent States — arXiv:2611.07552

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.07552
License (as archived)
CC0-1.0
Archived
2026-11-06T14:43:07.250Z
SHA-256 of the archived text
d501a6666cf481a00964c92a03f433b1ee3b8f7b9f099e96a4f99360ac51fff9
Stable URL of this page
https://metraipolis.com/documento/dc_46cd7378d37d

How to cite

SPARE: Fast Linear Attention Decoding with Compact Recurrent States — arXiv:2611.07552 — arXiv (en), https://arxiv.org/abs/2611.07552 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_46cd7378d37d (Metraipolis archive dc_46cd7378d37d, sha256 d501a6666cf481a00964c92a03f433b1ee3b8f7b9f099e96a4f99360ac51fff9)
BibTeX
@misc{dc46cd7378d37d,
  title = {SPARE: Fast Linear Attention Decoding with Compact Recurrent States — arXiv:2611.07552},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.07552},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_46cd7378d37d; sha256 d501a6666cf481a00964c92a03f433b1ee3b8f7b9f099e96a4f99360ac51fff9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07552. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Linear attention reduces the quadratic sequence-length complexity of softmax attention by maintaining fixed-size recurrent states. However, these states scale quadratically with the representation dimension and require repeated data transfers during decoding, creating a substantial GPU memory bottleneck. Recent studies have revealed significant low-rank redundancy in recurrent states, suggesting opportunities for more efficient representations. We introduce SPARE (State Projection for Accelerated Recurrent Execution), a training-free method that exploits this redundancy through fixed low-dimensional subspaces, heterogeneous rank allocation, and lightweight write compensation. By performing recurrent reads and updates directly on compact states, SPARE reduces memory traffic without online decomposition or dense-state reconstruction. SPARE achieves 2.55x full-model decoding speedup, 7.49x lower recurrent-kernel DRAM traffic, and 3.31x lower peak GPU memory, while preserving better associative-retrieval accuracy than structured pruning at matched memory budgets.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.07552 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:18:57.586Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).