Metraipolis › archive › dc_5dc19d30f23f

Towards Understanding Momentum Acceleration in River-Valley Loss Landscape — arXiv:2609.30957

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.30957
License (as archived)
CC0-1.0
Archived
2026-10-08T02:13:02.523Z
SHA-256 of the archived text
65f93747e613d09c6d440a53dfd5156cc5f65a21be8cd14e786031bb902c772f
Stable URL of this page
https://metraipolis.com/documento/dc_5dc19d30f23f

How to cite

Towards Understanding Momentum Acceleration in River-Valley Loss Landscape — arXiv:2609.30957 — arXiv (en), https://arxiv.org/abs/2609.30957 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5dc19d30f23f (Metraipolis archive dc_5dc19d30f23f, sha256 65f93747e613d09c6d440a53dfd5156cc5f65a21be8cd14e786031bb902c772f)
BibTeX
@misc{dc5dc19d30f23f,
  title = {Towards Understanding Momentum Acceleration in River-Valley Loss Landscape — arXiv:2609.30957},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.30957},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5dc19d30f23f; sha256 65f93747e613d09c6d440a53dfd5156cc5f65a21be8cd14e786031bb902c772f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.30957. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The empirical success of pretraining large language models has inspired a deeper investigation into the underlying loss landscapes and the optimization dynamics. Recent empirical and theoretical study suggest that the training loss landscape often exhibits a "river-valley" structure, which features a low-loss manifold (river) flanked by sharp orthogonal directions with higher loss (mountains). In the long term, the optimization progress is determined primarily by the progress along the river. Within such a landscape, gradient descent with large learning rates can move faster along the river despite high apparent loss due to vertical oscillations, while a subsequent sharp decay in the learning rate suppresses these oscillations, revealing genuine optimization progress. This explains the recent success of warmup-stable-decay (WSD) learning rate scheduler which, unlike cosine scheduling, keeps stable high learning rate and decays before producing intermediate checkpoints. Building on this foundation, in this work we take a step further and study the role of momentum within such a loss landscape. We establish theoretical analysis that characterizes how momentum accelerates optimization by stabilizing large learning rates that can not be tolerated by vanilla GD without deviating significantly from the river. The enabled large learning rate in-turn gives greater speed along the river and makes faster essential progress in the long run. Another intriguing observation from theory is that for a river-valley landscape with very flat and slow-spinning river, the momentum itself does not contribute directly to acceleration in terms of the speed of tracking the river, while the main acceleration comes from the admissible larger learning rate.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-28 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Optimization and Control (math.OC).
Pagina del lavoro: https://arxiv.org/abs/2609.30957 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T01:43:06.716Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).