Metraipolis › archive › dc_023488bebf27
Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States — arXiv:2610.20637
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.20637
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T10:19:53.370Z
- SHA-256 of the archived text
f35672667a7396dfd2ad717ac6e71bbf25da8bc19eaa04a0c9736b0858b1706e- Stable URL of this page
https://metraipolis.com/documento/dc_023488bebf27
How to cite
Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States — arXiv:2610.20637 — arXiv (en), https://arxiv.org/abs/2610.20637 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_023488bebf27 (Metraipolis archive dc_023488bebf27, sha256 f35672667a7396dfd2ad717ac6e71bbf25da8bc19eaa04a0c9736b0858b1706e)
BibTeX
@misc{dc023488bebf27,
title = {Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States — arXiv:2610.20637},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.20637},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_023488bebf27; sha256 f35672667a7396dfd2ad717ac6e71bbf25da8bc19eaa04a0c9736b0858b1706e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20637. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We expand the exponential kernel of attention into a truncated Taylor series of $n$ terms and factorize each term into a tensor-power inner product, which collapses the causal history into exactly $n$ recurrent states, each updated by one rank-one outer product per token and independent of context length $N$. The recursion is lossless with respect to the truncated kernel ($2.2 \times 10^{-16}$ in float64), and the states are additive across chunks ($5.7 \times 10^{-14}$), so prefill admits a parallel scan. Measured exponents are $t \propto N^{2.28}$ for softmax against $N^{0.91}$ for states, with a crossover at $N \approx 728$ and $20.8\times$ at $N = 8192$; per-token decoding latency is constant at $7.3\,\mu$s against $N^{1.04}$ for softmax, giving $76.0\times$ at $N = 65536$. Accuracy improves exponentially in $n$ while cost grows combinatorially, since the $m$-th state is $\binom{d+m-1}{m}$ wide---a bound confirmed by SVD---so the optimum is second order, $n = 3$, at $6.5 \times 10^{-4}$ error and $9{,}945$ elements per head. Query and key normalization is a prerequisite: without it the argument exceeds the convergence radius and adding terms stops helping.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.20637 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:48:02.637Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).