Metraipolis › archive › dc_2f54ff53d2ea

Spectral Degeneration of Self-Attention under Isotropic Score Geometry — arXiv:2611.02778

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.02778
License (as archived): CC0-1.0
Archived: 2026-11-03T10:58:39.863Z
SHA-256 of the archived text: a7504091e485685e51b5cf32c2981cb06f4e021afd759e2bd8798ec1e18337eb
Stable URL of this page: https://metraipolis.com/documento/dc_2f54ff53d2ea

How to cite

Spectral Degeneration of Self-Attention under Isotropic Score Geometry — arXiv:2611.02778 — arXiv (en), https://arxiv.org/abs/2611.02778 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2f54ff53d2ea (Metraipolis archive dc_2f54ff53d2ea, sha256 a7504091e485685e51b5cf32c2981cb06f4e021afd759e2bd8798ec1e18337eb)

BibTeX

@misc{dc2f54ff53d2ea,
  title = {Spectral Degeneration of Self-Attention under Isotropic Score Geometry — arXiv:2611.02778},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02778},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2f54ff53d2ea; sha256 a7504091e485685e51b5cf32c2981cb06f4e021afd759e2bd8798ec1e18337eb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02778. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We study the spectral structure of the self-attention matrix in Transformer models through an idealized spectral model for the pre-softmax score matrix. Motivated by the rotational isotropy of query and key representations, our model replaces the random token-space orientation of the score matrix by its canonical singular-value representative with a logarithmic spike--bulk separation. Empirical diagnostics on pretrained vision and language transformers motivate this model: token-space singular matrices have nearly uniform eigen-angle distributions, and score spectra are dominated by a small number of effective directions. We then prove that the row-wise softmax map transforms this structure into a degenerate attention matrix: dominant modes are preserved, whereas the bulk spectral distributions concentrate at zero. Stable rank remains bounded for bidirectional attention but grows logarithmically for causal attention. Within the canonical model, these results explain how attention can preserve a small number of dominant directions while compressing the remaining bulk modes.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2611.02778 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:49:38.680Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).