Metraipolis › archive › dc_69ff28041515
A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention — arXiv:2610.09051
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.09051
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T11:56:15.768Z
- SHA-256 of the archived text
897d6fc7cc1e6217932f6c2838a235fca2f84383b45eeec0f0512cbc1344857b- Stable URL of this page
https://metraipolis.com/documento/dc_69ff28041515
How to cite
A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention — arXiv:2610.09051 — arXiv (en), https://arxiv.org/abs/2610.09051 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_69ff28041515 (Metraipolis archive dc_69ff28041515, sha256 897d6fc7cc1e6217932f6c2838a235fca2f84383b45eeec0f0512cbc1344857b)
BibTeX
@misc{dc69ff28041515,
title = {A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention — arXiv:2610.09051},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.09051},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_69ff28041515; sha256 897d6fc7cc1e6217932f6c2838a235fca2f84383b45eeec0f0512cbc1344857b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09051. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The large KV-cache size of modern LLMs creates a barrier to efficient deployment. Recent work has explored replacing attention layers' RoPE positional embeddings with alternative decay-based mechanisms, which can then be used to prune KV-cache during inference. However, these decay functions have limited expressivity, and in practice devolve into sliding-window-like eviction patterns. In this work, we propose a unifying framework for complementary and novel decay mechanisms, capturing complex key statistics and interactions while preserving expressive RoPE embeddings and Softmax attention. The resulting Universal Attention is a highly expressive and end-to-end trainable architecture, whose composite decay mechanism acts as a natural, $\textit{adaptive}$ pruning criterion, removing tokens that contribute least to attention computation. Experimentally, Universal Attention achieves state-of-the-art $10\times$ compression on natural language and synthetic task data, while $\textit{improving}$ downstream performance compared to both state-of-the-art baselines and unpruned oracles. It further demonstrates superior long-context generalization with unprecedented $25\times$ compression at length 16k.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.09051 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:50:37.643Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).