Metraipolis › archive › dc_c0dead31a2ae

Emergent One-Third Scaling Law as Attention Tries to Concentrate — arXiv:2609.32100

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.32100
License (as archived)
CC0-1.0
Archived
2026-10-07T20:59:49.969Z
SHA-256 of the archived text
d297b3ba3b0373b8ce269539e728fe6956ea4bcf3ef47a0884201e65589ac26c
Stable URL of this page
https://metraipolis.com/documento/dc_c0dead31a2ae

How to cite

Emergent One-Third Scaling Law as Attention Tries to Concentrate — arXiv:2609.32100 — arXiv (en), https://arxiv.org/abs/2609.32100 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c0dead31a2ae (Metraipolis archive dc_c0dead31a2ae, sha256 d297b3ba3b0373b8ce269539e728fe6956ea4bcf3ef47a0884201e65589ac26c)
BibTeX
@misc{dcc0dead31a2ae,
  title = {Emergent One-Third Scaling Law as Attention Tries to Concentrate — arXiv:2609.32100},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32100},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c0dead31a2ae; sha256 d297b3ba3b0373b8ce269539e728fe6956ea4bcf3ef47a0884201e65589ac26c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32100. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The neural scaling law relating longer training to better performance through a power law is central to today's large language models (LLMs), yet its origin remains debated. One recent proposal is that power laws can emerge from the strong non-linearity of a single softmax head learning peaked distributions. What happens with multiple softmax functions, as in LLMs, is unclear. Here, we show through toy models that any softmax learning peaked distributions, regardless of its position in the model, can develop logit magnitudes that grow in a power law with exponent $1/3$, becoming a training bottleneck whose loss contribution decays as a power law with the same exponent $1/3$. The overall loss therefore obeys $1/3$ scaling whenever at least one softmax learns peaked distributions. We confirm that many softmax functions in LLMs learn peaked distributions and that LLM loss scaling matches this $1/3$ prediction. Moreover, logit growth dynamics reveal that attention heads, rather than the language modeling head, are the bottleneck likely driving the $1/3$ loss scaling in LLMs. Attention trying to concentrate on specific information, which is the heart of Transformers, may therefore also be the heart of the neural scaling law of training.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2609.32100 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:02.226Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).