Metraipolis › archive › dc_2b408102d96f
Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms — arXiv:2611.02943
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02943
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:38:47.473Z
- SHA-256 of the archived text
27bc0e7ef445f99e68a52e567ee8e41ff6cfcd7a162ac6a7e702bb29c0644aca- Stable URL of this page
https://metraipolis.com/documento/dc_2b408102d96f
How to cite
Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms — arXiv:2611.02943 — arXiv (en), https://arxiv.org/abs/2611.02943 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2b408102d96f (Metraipolis archive dc_2b408102d96f, sha256 27bc0e7ef445f99e68a52e567ee8e41ff6cfcd7a162ac6a7e702bb29c0644aca)
BibTeX
@misc{dc2b408102d96f,
title = {Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms — arXiv:2611.02943},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02943},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2b408102d96f; sha256 27bc0e7ef445f99e68a52e567ee8e41ff6cfcd7a162ac6a7e702bb29c0644aca},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02943. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Fusing independently normalized attention heads can increase the number of explicit key-value entries, or atoms, needed to preserve their output. We prove a sharp separation for $W_d(q)=\sum_{j=1}^d\sigma(q_j)$: its $d$ two-atom heads use $2d$ head-atom terms, whereas its minimum single-head representation has exactly $2^d$ atoms, even with arbitrary fixed replacement keys, scalar values, and positive masses. The same lower bound holds for every fixed uniform error below $1/2$ on $\mathbb R^d$, and the threshold is sharp. The proof uses exposed-key normal cones; exact equality on any nonempty open query domain inherits the bound. The resulting atom bound for a fixed head count is tight when that count divides $d$. A normalized family retains its exponential exact minimum while approaching a constant on fixed-radius query balls. For pretrained-model caches, a classical affine-rank bound and a certified instance exclude exact vector preservation at a specified atom count. Learned-cache comparisons on two disjoint banks of 64 contexts show budget-dependent reversals and calibration effects.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Neural and Evolutionary Computing (cs.NE); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2611.02943 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:12:54.603Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).