Metraipolis › archive › dc_07d8e873eccd
Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking — arXiv:2610.11206
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.11206
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T06:23:13.876Z
- SHA-256 of the archived text
f65e0835c4520f381793e711dbb8596e5aec77e5f9908f7c6d5507414e643cd2- Stable URL of this page
https://metraipolis.com/documento/dc_07d8e873eccd
How to cite
Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking — arXiv:2610.11206 — arXiv (en), https://arxiv.org/abs/2610.11206 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_07d8e873eccd (Metraipolis archive dc_07d8e873eccd, sha256 f65e0835c4520f381793e711dbb8596e5aec77e5f9908f7c6d5507414e643cd2)
BibTeX
@misc{dc07d8e873eccd,
title = {Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking — arXiv:2610.11206},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.11206},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_07d8e873eccd; sha256 f65e0835c4520f381793e711dbb8596e5aec77e5f9908f7c6d5507414e643cd2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11206. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Flat loss landscapes have long been linked to better generalization in neural networks. However, its role as a causal mechanism for generalization is less established. Grokking provides an unique testbed to understand this distinction: models are prone to fit observed data using non-generalizing structure and remain in that regime for prolonged periods, transitioning to generalization only under particular training conditions. In this work, we study whether flat loss landscapes can act as a driving mechanism in this transition. While recent work has argued for flatness as a necessary geometric condition for this transition, we find that biasing training toward flatter solutions using sharpness-aware minimization (SAM) is insufficient to reliably induce this transition, despite producing flatter solutions. However, when SAM is paired with mechanisms that drive generalization such as weight decay, an interesting property emerges: SAM can accelerate the transition to generalizing solutions by up to 4x at the epoch-level. We theoretically untangle this relationship between SAM and weight decay using a minimal interpolating two-layer ReLU model with both memorizing and generalizing solutions. We show that even in this simple setup, flatness alone cannot distinguish a memorizing solution from a generalizing one, while weight decay favors generalizing solutions. However, under a local stability analysis, there exists a window where a memorizing interpolant is locally stable under gradient descent but unstable under SAM in the low-norm regime, which can explain SAM's ability to accelerate this transition. Overall, our results provide a more interpretable account of the role of flatness in driving generalization, especially in settings where models are vulnerable to minimizing loss through learning non-generalizing structure. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.11206 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T05:50:31.193Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).