Metraipolis › archive › dc_c031d05a044f
Kernel-based Interpolatory Methods for Context Compression of Attention Mechanisms — arXiv:2610.40943
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.40943
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T08:02:34.003Z
- SHA-256 of the archived text
c426a5f6ed94d75dd3122cdb00c0e672d3ba32f07fc4a551240d453a637a8382- Stable URL of this page
https://metraipolis.com/documento/dc_c031d05a044f
How to cite
Kernel-based Interpolatory Methods for Context Compression of Attention Mechanisms — arXiv:2610.40943 — arXiv (en), https://arxiv.org/abs/2610.40943 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c031d05a044f (Metraipolis archive dc_c031d05a044f, sha256 c426a5f6ed94d75dd3122cdb00c0e672d3ba32f07fc4a551240d453a637a8382)
BibTeX
@misc{dcc031d05a044f,
title = {Kernel-based Interpolatory Methods for Context Compression of Attention Mechanisms — arXiv:2610.40943},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.40943},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c031d05a044f; sha256 c426a5f6ed94d75dd3122cdb00c0e672d3ba32f07fc4a551240d453a637a8382},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40943. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The bottleneck of attention mechanisms, which are the fundamental building block of modern language models, is their quadratic complexity in terms of context size. There has thus been an increasing interest in the compression of the context, also called KV-cache compression. In this article, we consider the problem of compressing the context of attention mechanisms by first formulating it in the Hilbert space associated with its kernel. In this formalism, attention can be represented as the ratio of two linear maps whose interpolant is also the ratio of interpolants, this allows us to consider the interpolation of the linear maps directly. We thus introduce two novel KV-cache compression methods which we call GAIM and Q-DAIM, based on the Generalized Empirical Interpolation Method (GEIM) and the Q-Discrete Empirical Interpolation Method (Q-DEIM), respectively. As the cost of these methods is quadratic in terms of context length, we propose surrogate models based on the Taylor series, the Performer and the Random Fourier Features method which allow the compression to be done in linear time. Finally, we extend this result to the streaming attention setting, providing new methods which allow one to penalize the selection of interpolation nodes based on the previously interpolated context and to update the previous interpolation based on new information. We then provide numerical results on the convergence of the residuals and the output error in future queries and preliminary results on the LongBench-E benchmark showing performance comparable to the state-of-the-art methods. These experiments suggest that the range revealed by our Q-DAIM method could help identify where information is concentrated in the KV cache, thus enabling an efficient compression. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Numerical Analysis (math.NA). Pagina del lavoro: https://arxiv.org/abs/2610.40943 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T07:59:38.774Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).