Metraipolis › archive › dc_445bee06c205

Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizons — arXiv:2609.35029

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.35029
License (as archived)
CC0-1.0
Archived
2026-10-07T22:38:44.490Z
SHA-256 of the archived text
67a25e35f33e7013bb5f8dd5d5e90eddb9a538e1db3fdf87d5c4e7dfb207ed04
Stable URL of this page
https://metraipolis.com/documento/dc_445bee06c205

How to cite

Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizons — arXiv:2609.35029 — arXiv (en), https://arxiv.org/abs/2609.35029 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_445bee06c205 (Metraipolis archive dc_445bee06c205, sha256 67a25e35f33e7013bb5f8dd5d5e90eddb9a538e1db3fdf87d5c4e7dfb207ed04)
BibTeX
@misc{dc445bee06c205,
  title = {Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizons — arXiv:2609.35029},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35029},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_445bee06c205; sha256 67a25e35f33e7013bb5f8dd5d5e90eddb9a538e1db3fdf87d5c4e7dfb207ed04},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35029. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Hyperparameter transfer across model width can substantially reduce the cost of tuning large neural networks, but its behavior when the training horizon grows with width is not fully understood. Building on the framework of fast hyperparameter transfer (Ghosh et al., 2026), which formalizes when transfer is effective, we investigate conditions that ensure fast transfer in the growing-horizon regime. Specifically, we study learning-rate transfer in a shallow linear network with a single trainable hidden matrix, trained by full-batch gradient descent. Under additional spectral assumptions, our main results are threefold. (i) We prove fast learning-rate transfer as $n,T\to\infty$ whenever $T=o(\sqrt{n})$. (ii) We characterize the transfer rates through the finite-width perturbation scale, the first-order sensitivities of the loss and its learning-rate derivative to finite-width perturbations, and the local loss curvature. (iii) We derive limiting distributions for the optimal learning rate and optimized loss, governed by fluctuations associated with the extreme eigenvalues of the data Gram matrix. These results clarify how spectral structure and local loss sensitivities govern learning-rate transfer at growing horizons.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2609.35029 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:28.329Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).