Metraipolis › archive › dc_5055c331d634

Early learning in large residual networks — arXiv:2610.31783

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.31783
License (as archived): CC0-1.0
Archived: 2026-10-26T20:13:54.058Z
SHA-256 of the archived text: 85391423378aab818c390ead0bbb250eeabaae4dd98ea22dbf9228644bc5c0ec
Stable URL of this page: https://metraipolis.com/documento/dc_5055c331d634

How to cite

Early learning in large residual networks — arXiv:2610.31783 — arXiv (en), https://arxiv.org/abs/2610.31783 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5055c331d634 (Metraipolis archive dc_5055c331d634, sha256 85391423378aab818c390ead0bbb250eeabaae4dd98ea22dbf9228644bc5c0ec)

BibTeX

@misc{dc5055c331d634,
  title = {Early learning in large residual networks — arXiv:2610.31783},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.31783},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5055c331d634; sha256 85391423378aab818c390ead0bbb250eeabaae4dd98ea22dbf9228644bc5c0ec},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31783. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We study the early training dynamics of nonlinear residual networks with blocks of zero hidden layers as a model of more general residual networks. We analyze networks in a sequential limit where first width and depth diverge, then the number of training data points diverges, and finally the input dimension diverges. This is a more refined setup compared to the typical mean-field theory analysis in which depth and width diverge but the number of training datapoints and input dimension are fixed. Our main technical result is a diagrammatic computation of both the network output and the hidden representations after the few steps of gradient descent in terms of the statistics of inner products of inputs from the training set. One of our principle conceptual findings is a negative result: in the limit of large input dimension the network output can learn only a linear function of its inputs early in training since higher learned monomials are more and more suppressed in the input dimension and vanish in the large dimension limit. This suggests that mean-field limits of ResNets are surprisingly rigid in the regime of high input dimension and may not be sufficient toy models for understanding feature learning. Finally, we investigate how to ensure hyperparameter transfer for the learning rate as a function of dataset size and input dimension. We test our predictions numerically and find excellent agreement with our theory.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (stat.ML) ; Machine Learning (cs.LG); Probability (math.PR); Data Analysis, Statistics and Probability (physics.data-an).
Pagina del lavoro: https://arxiv.org/abs/2610.31783 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-26T19:49:03.452Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).