Metraipolis › archive › dc_bd4be90ec126
On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds — arXiv:2608.03197
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.03197
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T12:42:22.512Z
- SHA-256 of the archived text
16c6e68e9bbbe27b2b09b1e25e3c5d671c2ed35f470aa1e3e26c0377b6be5c75- Stable URL of this page
https://metraipolis.com/documento/dc_bd4be90ec126
How to cite
On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds — arXiv:2608.03197 — arXiv (en), https://arxiv.org/abs/2608.03197 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bd4be90ec126 (Metraipolis archive dc_bd4be90ec126, sha256 16c6e68e9bbbe27b2b09b1e25e3c5d671c2ed35f470aa1e3e26c0377b6be5c75)
BibTeX
@misc{dcbd4be90ec126,
title = {On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds — arXiv:2608.03197},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.03197},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bd4be90ec126; sha256 16c6e68e9bbbe27b2b09b1e25e3c5d671c2ed35f470aa1e3e26c0377b6be5c75},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03197. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Sharpness-Aware Minimization (SAM) improves generalization by seeking parameters whose loss is robust to local adversarial perturbations, but the quantitative mechanism underlying its implicit bias toward flat minima remains unclear. In particular, the perturbation radius $\rho$ is typically treated as an isolated tuning parameter, despite defining the neighborhood in which SAM measures sharpness. We analyze mini-batch SAM near an interpolating minimum through linear stability. Under local linearization and gradient-noise alignment assumptions, we prove that every linearly stable minimum satisfies $\lambda_{\max}\leq\sqrt[3]{b\Gamma/(2\rho\eta^2)}$, where $\lambda_{\max}$ is the largest Hessian eigenvalue, $b$ is the batch size, $\eta$ is the learning rate, and $\Gamma$ bounds the gradient norm. The bound quantitatively characterizes SAM's implicit flatness bias: holding the other quantities fixed, a smaller batch size, a larger learning rate, or a larger radius restricts linearly stable SAM to flatter minima. It also exposes a necessary trade-off: $\rho$ should be large enough to promote flatness, yet remain local enough to preserve the approximation and stable training. We validate this prediction in a controlled study of 900 models on CIFAR-100 with ResNet-18 and VGG-19, where increasing $\rho$ is consistently associated with a smaller largest Hessian eigenvalue across batch-size and learning-rate settings. Finally, we instantiate the analysis in Taylor-Locality Controlled SAM (TLC-SAM), which adjusts $\rho$ using the observed Taylor-approximation error and further reduces the top Hessian eigenvalue relative to fixed-radius SAM. Our results provide quantitative hyperparameter bounds and a stability--locality perspective for analyzing and designing SAM variants.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.03197 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:31:30.343Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).