Metraipolis › archive › dc_8f7b44facd38

Spectral Degeneration of Self-Attention under Isotropic Score Geometry — arXiv:2611.02778

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.02778
License (as archived)
CC0-1.0
Archived
2026-11-03T18:20:09.526Z
SHA-256 of the archived text
ad833710af5f3daec6732a75350acc9822265ba57386ba25c2ea0ee863836753
Stable URL of this page
https://metraipolis.com/documento/dc_8f7b44facd38

How to cite

Spectral Degeneration of Self-Attention under Isotropic Score Geometry — arXiv:2611.02778 — arXiv (en), https://arxiv.org/abs/2611.02778 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8f7b44facd38 (Metraipolis archive dc_8f7b44facd38, sha256 ad833710af5f3daec6732a75350acc9822265ba57386ba25c2ea0ee863836753)
BibTeX
@misc{dc8f7b44facd38,
  title = {Spectral Degeneration of Self-Attention under Isotropic Score Geometry — arXiv:2611.02778},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02778},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8f7b44facd38; sha256 ad833710af5f3daec6732a75350acc9822265ba57386ba25c2ea0ee863836753},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02778. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We study the spectral structure of the self-attention matrix in Transformer models through an idealized spectral model for the pre-softmax score matrix. Motivated by the rotational isotropy of query and key representations, our model replaces the random token-space orientation of the score matrix by its canonical singular-value representative with a logarithmic spike--bulk separation. Empirical diagnostics on pretrained vision and language transformers motivate this model: token-space singular matrices have nearly uniform eigen-angle distributions, and score spectra are dominated by a small number of effective directions. We then prove that the row-wise softmax map transforms this structure into a degenerate attention matrix: dominant modes are preserved, whereas the bulk spectral distributions concentrate at zero. Stable rank remains bounded for bidirectional attention but grows logarithmically for causal attention. Within the canonical model, these results explain how attention can preserve a small number of dominant directions while compressing the remaining bulk modes.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2611.02778 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:09:53.713Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).