Metraipolis › archive › dc_6f181bca82c0

Emergent Inverse-Depth Scaling From Nonlinearity In Attention — arXiv:2610.11063

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.11063
License (as archived)
CC0-1.0
Archived
2026-10-09T05:57:01.408Z
SHA-256 of the archived text
dcce4c8dd65f07f131e6eb8e04b9cb2730c493f836aa412d4db879493885bad3
Stable URL of this page
https://metraipolis.com/documento/dc_6f181bca82c0

How to cite

Emergent Inverse-Depth Scaling From Nonlinearity In Attention — arXiv:2610.11063 — arXiv (en), https://arxiv.org/abs/2610.11063 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6f181bca82c0 (Metraipolis archive dc_6f181bca82c0, sha256 dcce4c8dd65f07f131e6eb8e04b9cb2730c493f836aa412d4db879493885bad3)
BibTeX
@misc{dc6f181bca82c0,
  title = {Emergent Inverse-Depth Scaling From Nonlinearity In Attention — arXiv:2610.11063},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.11063},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6f181bca82c0; sha256 dcce4c8dd65f07f131e6eb8e04b9cb2730c493f836aa412d4db879493885bad3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11063. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Scaling laws describe power-law improvements in model performance with dataset size and parameter count, yet their underlying mechanisms are not fully understood. To explain the parameter count scaling, existing theory posits power-law scaling with model depth. In linear-attention models, this scaling is tied to a power-law data spectrum: unable to selectively attend to relevant tokens, these models learn according to global spectral strength, with stronger directions learned before weaker ones. Large language models, however, can be strongly nonlinear. Here, we show that nonlinear attention yields inverse-depth decay of loss across all tested data spectra. Nonlinearity enables attention to focus selectively on relevant tokens, allowing strong and weak spectral directions to be learned in parallel. Similar focusing across layers motivates a connection to the central limit theorem: shared error across layers sets the loss plateau, while aggregation turns layer-specific differences into continued gains with depth. Our findings suggest that depth scaling may arise from nonlinearity in attention, which allows large language models to focus locally and may make the global covariance structure less relevant.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.11063 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T05:47:01.500Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).