Metraipolis › archive › dc_10be55208fb0

Infinite ResNets are Gaussian Mean ODEs — arXiv:2610.41514

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.41514
License (as archived)
CC0-1.0
Archived
2026-11-03T08:11:56.477Z
SHA-256 of the archived text
6415acc4709f8692b70fea2f123daf01ca59a19670045e54628832f485b90169
Stable URL of this page
https://metraipolis.com/documento/dc_10be55208fb0

How to cite

Infinite ResNets are Gaussian Mean ODEs — arXiv:2610.41514 — arXiv (en), https://arxiv.org/abs/2610.41514 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_10be55208fb0 (Metraipolis archive dc_10be55208fb0, sha256 6415acc4709f8692b70fea2f123daf01ca59a19670045e54628832f485b90169)
BibTeX
@misc{dc10be55208fb0,
  title = {Infinite ResNets are Gaussian Mean ODEs — arXiv:2610.41514},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41514},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_10be55208fb0; sha256 6415acc4709f8692b70fea2f123daf01ca59a19670045e54628832f485b90169},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41514. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Understanding the training dynamics of deep neural networks in large-scale limits is a question both of foundational and of practical importance. Prior works have obtained such descriptions through dynamical mean field theory (DMFT), yielding a system of non-Markovian dynamics that are challenging to analyze and that obscure the optimization structure of the original dynamics. In this paper, we obtain instead a simple description of the large-scale limit training dynamics for ResNets in standard shape scalings and in the $1/\text{depth}$ branch scaling that allows maximal (local) feature updates. For a model trained with gradient descent (GD) from a random initialization, the limit dynamics is simply GD on a Neural Mean ODE with Gaussian initial weights. Our analysis covers population or empirical risk GD in the joint limit of large width, depth and embedding, input, and output dimensions. We obtain approximation scaling laws that depend on the shape of the ResNet and the covariance spectrum of the data. We also illustrate the flexibility of our approach by discussing Mixture of Expert models, where likewise the limit admits an intuitive (nested) Mean ODE description.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.41514 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T08:02:38.929Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).