Metraipolis › archive › dc_009061b2aa8f
Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizons — arXiv:2609.33041
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33041
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T22:37:06.456Z
- SHA-256 of the archived text
7cba8133e405d4ca9accaa9ff001fd324ecd362b78eb13420e03c432e761767e- Stable URL of this page
https://metraipolis.com/documento/dc_009061b2aa8f
How to cite
Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizons — arXiv:2609.33041 — arXiv (en), https://arxiv.org/abs/2609.33041 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_009061b2aa8f (Metraipolis archive dc_009061b2aa8f, sha256 7cba8133e405d4ca9accaa9ff001fd324ecd362b78eb13420e03c432e761767e)
BibTeX
@misc{dc009061b2aa8f,
title = {Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizons — arXiv:2609.33041},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33041},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_009061b2aa8f; sha256 7cba8133e405d4ca9accaa9ff001fd324ecd362b78eb13420e03c432e761767e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33041. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Learning-rate warmup is a standard technique in language-model training, yet its duration remains largely heuristic. Common approaches use either a fixed number of updates or a fixed fraction of the training horizon, two choices that imply very different scaling as training gets longer. When should warmup stay fixed, and when should it grow with the horizon? We address this question with a quadratic model whose modes respond differently to the peak learning rate. Warmup slows progress in directions that already contract well at the peak rate, but can remove persistent error in directions near the stability edge, with higher peak rates shifting the balance toward longer warmup durations. This yields a compact horizon scaling law that captures regimes ranging from essentially no warmup, through fixed-duration warmup, to durations that grow with the training horizon, and explains how the preferred regime changes with peak learning rate. Because the law captures the tradeoff between giving up early progress and improving the trajectory that follows, it can be fit using shorter runs and used to predict warmup at substantially longer horizons. Together, our results explain several familiar properties of warmup through a single tradeoff and suggest treating warmup duration as a horizon-dependent hyperparameter rather than a fixed training heuristic. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.33041 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T22:14:27.845Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).