Metraipolis › archive › dc_652873cc592d
Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining — arXiv:2608.24814
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.24814
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T07:00:19.502Z
- SHA-256 of the archived text
52bad24cf6d6fc65d0aa038fcf46f8f6ef32b0a0c1bbdf17379f3c9857218ae8- Stable URL of this page
https://metraipolis.com/documento/dc_652873cc592d
How to cite
Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining — arXiv:2608.24814 — arXiv (en), https://arxiv.org/abs/2608.24814 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_652873cc592d (Metraipolis archive dc_652873cc592d, sha256 52bad24cf6d6fc65d0aa038fcf46f8f6ef32b0a0c1bbdf17379f3c9857218ae8)
BibTeX
@misc{dc652873cc592d,
title = {Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining — arXiv:2608.24814},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.24814},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_652873cc592d; sha256 52bad24cf6d6fc65d0aa038fcf46f8f6ef32b0a0c1bbdf17379f3c9857218ae8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.24814. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods. The resulting ELR-based FSL also explains delayed acceleration, a recurring effect of norm control. Together, these results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics. Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-02 in stat.ML). Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML). Pagina del lavoro: https://arxiv.org/abs/2608.24814 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T06:47:06.996Z dalla pagina https://arxiv.org/catchup/stat.ML/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).