Metraipolis › archive › dc_7ecc2393824a

Exploiting Biases in Attention for Transformers — arXiv:2611.04657

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04657
License (as archived)
CC0-1.0
Archived
2026-11-04T18:54:09.669Z
SHA-256 of the archived text
b623c9033e8a3a3f9a3d6306c64dfb26d2bc69dd47cc2c0d6d45d7425658e508
Stable URL of this page
https://metraipolis.com/documento/dc_7ecc2393824a

How to cite

Exploiting Biases in Attention for Transformers — arXiv:2611.04657 — arXiv (en), https://arxiv.org/abs/2611.04657 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7ecc2393824a (Metraipolis archive dc_7ecc2393824a, sha256 b623c9033e8a3a3f9a3d6306c64dfb26d2bc69dd47cc2c0d6d45d7425658e508)
BibTeX
@misc{dc7ecc2393824a,
  title = {Exploiting Biases in Attention for Transformers — arXiv:2611.04657},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04657},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7ecc2393824a; sha256 b623c9033e8a3a3f9a3d6306c64dfb26d2bc69dd47cc2c0d6d45d7425658e508},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04657. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Attention is known to be sparse in vision transformers. In particular, we analyze the unequal relevance of attention-matrix columns to the final computation and show that this phenomenon becomes increasingly pronounced with network depth. This can be interpreted as saying that tokens have an absolute importance for the attention process, independently of queries. Inspired by this, we propose a reformulation of the attention formula that makes this phenomenon explicit by adding an input-dependent bias to the query-key dot product. We show that this new term gives each token a scalar value indicating its anticipated influence on other tokens during the attention process, and can thus serve as a proxy for pruning attention maps. Across multiple transformer model sizes, this allows us to substantially reduce the computational cost of attention, in terms of both FLOPs and inference latency, while retaining most of the original classification accuracy. We publicly released the code and models at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.04657 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:35:37.447Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).