Metraipolis › archive › dc_ad9ee283a763
Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training — arXiv:2610.40362
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.40362
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-01T02:25:24.894Z
- SHA-256 of the archived text
7df77f94b3549c46d424df88e52504d1f27ad85e1b18cfe89035aacaf245d88e- Stable URL of this page
https://metraipolis.com/documento/dc_ad9ee283a763
How to cite
Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training — arXiv:2610.40362 — arXiv (en), https://arxiv.org/abs/2610.40362 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ad9ee283a763 (Metraipolis archive dc_ad9ee283a763, sha256 7df77f94b3549c46d424df88e52504d1f27ad85e1b18cfe89035aacaf245d88e)
BibTeX
@misc{dcad9ee283a763,
title = {Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training — arXiv:2610.40362},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.40362},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ad9ee283a763; sha256 7df77f94b3549c46d424df88e52504d1f27ad85e1b18cfe89035aacaf245d88e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40362. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Weight-normalized (WN) training is emerging as a compelling alternative to conventional weight decay (WD), with recent studies reporting substantial pretraining loss gains. We show that these gains are driven primarily by the induced effective learning rate (ELR) trajectory rather than by normalization alone. Matching ELR trajectories largely collapses the training loss dynamics of WN and WD. Schedule design should therefore target ELR rather than the nominal learning rate (LR) alone. Building on this principle, we propose AIR (Affine Inverse-square-Root decay), which combines a theoretically derived $1/\sqrt{t}$ decay with an affine adjustment to satisfy prescribed peak and terminal LRs over a finite training horizon. We further identify late-stage ELR scheduling as a key source of its gains, because a shared late schedule attenuates the effect of earlier LR choices. Experiments on 128 B-token Mixture-of-Experts pretraining demonstrate that AIR outperforms cosine and other candidate schedules in training loss, downstream performance, and hyperparameter robustness, reducing terminal training loss by approximately 0.01 relative to cosine at matched cumulative LR. Applying the same ELR trajectory to WD reproduces the corresponding training dynamics, showing that the benefits extend beyond fixed-norm training and establishing ELR as the principled scheduling object.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.40362 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-01T02:12:44.209Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).