Metraipolis › archive › dc_fa585c8fd9ea

The Road Taken: The Role of Optimizers at the Edge of Stability — arXiv:2608.18415

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.18415
License (as archived): CC0-1.0
Archived: 2026-10-18T13:53:13.665Z
SHA-256 of the archived text: 23f7fb3556d5e8ffabe10444e0117f9c7dda3bec5cf2fb0aa21372344a7bcb57
Stable URL of this page: https://metraipolis.com/documento/dc_fa585c8fd9ea

How to cite

The Road Taken: The Role of Optimizers at the Edge of Stability — arXiv:2608.18415 — arXiv (en), https://arxiv.org/abs/2608.18415 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fa585c8fd9ea (Metraipolis archive dc_fa585c8fd9ea, sha256 23f7fb3556d5e8ffabe10444e0117f9c7dda3bec5cf2fb0aa21372344a7bcb57)

BibTeX

@misc{dcfa585c8fd9ea,
  title = {The Road Taken: The Role of Optimizers at the Edge of Stability — arXiv:2608.18415},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.18415},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fa585c8fd9ea; sha256 23f7fb3556d5e8ffabe10444e0117f9c7dda3bec5cf2fb0aa21372344a7bcb57},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.18415. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The edge of stability refers to a phenomenon in deep learning with gradient-based optimizers where the Hessian eigenvalues of the loss remain stable above a threshold that the classical descent lemma predicts to be unstable. Previous works formulate the edge of stability with respect to the maximum Hessian eigenvalue and the learning rate. However, we observe that many first-order methods, including gradient descent, significantly violate the stability bound predicted by these theories by a factor as large as $\times 21.1$. Moreover, this deviation turns out to be systematic and highly dependent on the underlying optimizer, which is not captured by previous formulations. This calls for a new formulation of the stability threshold, which we derive from the directional Hessian and the gradient-alignment score with respect to the actual update taken by the optimizer, rather than the maximum curvature mode. Our new formulation of the realized edge of stability not only removes optimizer-dependent offsets and provides more consistent predictions of the stability threshold, but also introduces new diagnostic tools that reveal the unique role of the optimizer in actively balancing between the temporal and spatial budgets in first-order optimization.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-20 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.18415 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T13:50:58.904Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).