Metraipolis › archive › dc_0b8ed151ce73
NS-ATTENTION: Newton-Schulz Transformations of Attention Outputs in Vision Transformers — arXiv:2609.27735
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.27735
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T10:57:18.351Z
- SHA-256 of the archived text
2ae404e5e3fccef021516caa9aac93c51e70e97fa27e80ae6dc80451a4e8b648- Stable URL of this page
https://metraipolis.com/documento/dc_0b8ed151ce73
How to cite
NS-ATTENTION: Newton-Schulz Transformations of Attention Outputs in Vision Transformers — arXiv:2609.27735 — arXiv (en), https://arxiv.org/abs/2609.27735 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0b8ed151ce73 (Metraipolis archive dc_0b8ed151ce73, sha256 2ae404e5e3fccef021516caa9aac93c51e70e97fa27e80ae6dc80451a4e8b648)
BibTeX
@misc{dc0b8ed151ce73,
title = {NS-ATTENTION: Newton-Schulz Transformations of Attention Outputs in Vision Transformers — arXiv:2609.27735},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.27735},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0b8ed151ce73; sha256 2ae404e5e3fccef021516caa9aac93c51e70e97fa27e80ae6dc80451a4e8b648},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.27735. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Newton-Schulz (NS) iteration has recently been used in the Muon optimizer to transform update matrices during the training of large language models. Motivated by its spectral effect, we investigate applying NS directly to Transformer attention representations. We introduce Newton-Schulz Attention (NS-Attn.), a parameter-free transformation applied to the output of each attention head. Each head output is arranged as a feature-by-token matrix and normalized by its Frobenius norm. We then apply a finite NS polynomial step and restore the original norm. The objective is to reduce spectral concentration and increase effective rank before standard head merging and output projection. Across ViT and Swin on CIFAR-10 and CIFAR-100, NS-Attn. improves final-epoch accuracy in all 12 matched-seed comparisons, with mean gains of 0.25--0.83 percentage points. ViT ablations show higher mean accuracy with one iteration than with two. Spectral analysis further shows reduced leading-eigenvalue concentration and increased effective rank. These gains incur additional inference latency. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.27735 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T10:06:16.744Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).