Metraipolis › archive › dc_46cd7378d37d
SPARE: Fast Linear Attention Decoding with Compact Recurrent States — arXiv:2611.07552
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.07552
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-06T14:43:07.250Z
- SHA-256 of the archived text
d501a6666cf481a00964c92a03f433b1ee3b8f7b9f099e96a4f99360ac51fff9- Stable URL of this page
https://metraipolis.com/documento/dc_46cd7378d37d
How to cite
SPARE: Fast Linear Attention Decoding with Compact Recurrent States — arXiv:2611.07552 — arXiv (en), https://arxiv.org/abs/2611.07552 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_46cd7378d37d (Metraipolis archive dc_46cd7378d37d, sha256 d501a6666cf481a00964c92a03f433b1ee3b8f7b9f099e96a4f99360ac51fff9)
BibTeX
@misc{dc46cd7378d37d,
title = {SPARE: Fast Linear Attention Decoding with Compact Recurrent States — arXiv:2611.07552},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.07552},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_46cd7378d37d; sha256 d501a6666cf481a00964c92a03f433b1ee3b8f7b9f099e96a4f99360ac51fff9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07552. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Linear attention reduces the quadratic sequence-length complexity of softmax attention by maintaining fixed-size recurrent states. However, these states scale quadratically with the representation dimension and require repeated data transfers during decoding, creating a substantial GPU memory bottleneck. Recent studies have revealed significant low-rank redundancy in recurrent states, suggesting opportunities for more efficient representations. We introduce SPARE (State Projection for Accelerated Recurrent Execution), a training-free method that exploits this redundancy through fixed low-dimensional subspaces, heterogeneous rank allocation, and lightweight write compensation. By performing recurrent reads and updates directly on compact states, SPARE reduces memory traffic without online decomposition or dense-state reconstruction. SPARE achieves 2.55x full-model decoding speedup, 7.49x lower recurrent-kernel DRAM traffic, and 3.31x lower peak GPU memory, while preserving better associative-retrieval accuracy than structured pruning at matched memory budgets. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2611.07552 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-06T14:18:57.586Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).