Metraipolis › archive › dc_7d7f0e9fc02b
A Journey to the Edge of Stability — arXiv:2609.32290
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.32290
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T22:36:26.872Z
- SHA-256 of the archived text
d53061886f20249a89f1e2e2e1e3d153dd036b0f8222dd5868fb4003c7339580- Stable URL of this page
https://metraipolis.com/documento/dc_7d7f0e9fc02b
How to cite
A Journey to the Edge of Stability — arXiv:2609.32290 — arXiv (en), https://arxiv.org/abs/2609.32290 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7d7f0e9fc02b (Metraipolis archive dc_7d7f0e9fc02b, sha256 d53061886f20249a89f1e2e2e1e3d153dd036b0f8222dd5868fb4003c7339580)
BibTeX
@misc{dc7d7f0e9fc02b,
title = {A Journey to the Edge of Stability — arXiv:2609.32290},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.32290},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7d7f0e9fc02b; sha256 d53061886f20249a89f1e2e2e1e3d153dd036b0f8222dd5868fb4003c7339580},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32290. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
It has recently been found that deep learning often occurs at the "edge of stability (EoS)," where the maximum Hessian eigenvalue of the model is stabilized at a value reciprocal to the learning rate. However, what happens before we reach that regime? We fix a deep learning problem and vary first order optimization methods with dense learning rate sweeps. We then track the characterizing quantities of a learning trajectory: the loss, the sharpness, and the alignment between consecutive gradients. To our surprise, if we scale the learning rate by the dc gain of the optimizer, these traces from the sweeps from different optimizers almost perfectly overlap across a large range of learning rates. The dc-normalized optimizers have another role that only becomes apparent in high learning rates: they select when the sharpness value detaches from this universal curve and enters the edge of stability. Upon this discovery, we specify three distinct regimes with respect to the dc-adjusted learning rate: the low-LR regime where the trajectory is nearly insensitive to the optimizer, the high-LR regime, where the optimizer governs the sharpness according to the EoS reciprocal rule, and the in-between mid-LR regime where so-called progressive sharpening originates independently of the optimizer. This distinguishes the role of the optimizer, the learning rate, and the model in shaping the learning progress. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.32290 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T22:14:27.643Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).