Metraipolis › archive › dc_448cb03e7965
Misalignment of Low-Loss Regions Causes Grokking — arXiv:2610.00620
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.00620
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T12:10:30.034Z
- SHA-256 of the archived text
84d448224b47e77248fcf8c105c389356ea376827a9b5445949bae7063912a2f- Stable URL of this page
https://metraipolis.com/documento/dc_448cb03e7965
How to cite
Misalignment of Low-Loss Regions Causes Grokking — arXiv:2610.00620 — arXiv (en), https://arxiv.org/abs/2610.00620 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_448cb03e7965 (Metraipolis archive dc_448cb03e7965, sha256 84d448224b47e77248fcf8c105c389356ea376827a9b5445949bae7063912a2f)
BibTeX
@misc{dc448cb03e7965,
title = {Misalignment of Low-Loss Regions Causes Grokking — arXiv:2610.00620},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.00620},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_448cb03e7965; sha256 84d448224b47e77248fcf8c105c389356ea376827a9b5445949bae7063912a2f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00620. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Grokking refers to the delayed emergence of validation-set generalization after a model has already overfit the training set. Although first observed in small algorithmic tasks trained with transformers, its underlying mechanism remains unsettled. In this work, we develop an analysis framework based on mode connectivity and the geometry of low-loss regions. The framework predicts that the standard modular-arithmetic setting does not always produce grokking: under a symmetry-preserving train/validation split, we observe a stable anti-grokking case in which validation performance does not recover. This counterexample challenges several existing correlational explanations of grokking. More broadly, our analysis framework and results further suggest that grokking arises when the low-loss regions induced by the training and validation partitions are misaligned. Once these regions become well aligned, training hyperparameters alone cannot produce grokking and the observed dynamics collapse to either trainable or non-trainable behavior. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.00620 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T10:21:28.095Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).