Metraipolis › archive › dc_07b20330561b
On Lipschitz Explosion in Deep Neural Networks with Normalization: Consequences for Optimization and Adversarial Robustness — arXiv:2611.08775
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.08775
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-09T22:34:34.857Z
- SHA-256 of the archived text
2750b1d1cec51c6fe4ec2360c7001c4b2fe03552beb043d62660db5cd01ae837- Stable URL of this page
https://metraipolis.com/documento/dc_07b20330561b
How to cite
On Lipschitz Explosion in Deep Neural Networks with Normalization: Consequences for Optimization and Adversarial Robustness — arXiv:2611.08775 — arXiv (en), https://arxiv.org/abs/2611.08775 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_07b20330561b (Metraipolis archive dc_07b20330561b, sha256 2750b1d1cec51c6fe4ec2360c7001c4b2fe03552beb043d62660db5cd01ae837)
BibTeX
@misc{dc07b20330561b,
title = {On Lipschitz Explosion in Deep Neural Networks with Normalization: Consequences for Optimization and Adversarial Robustness — arXiv:2611.08775},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.08775},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_07b20330561b; sha256 2750b1d1cec51c6fe4ec2360c7001c4b2fe03552beb043d62660db5cd01ae837},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.08775. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Can a network look stable at every layer and still be exponentially unstable as a whole? The Lipschitz constant of a neural network is a basic stability quantity appearing in optimization guarantees, generalization bounds, and robustness certificates. Deep networks naturally admit two such notions, input Lipschitzness, measuring sensitivity to data perturbations, and parameter Lipschitzness, measuring sensitivity to weight perturbations. We prove that, for deep networks with normalization layers, with batch normalization as the canonical example, both Lipschitz constants can grow exponentially with depth and hence with parameter dimension, even under strong per-layer norm control such as $\|W_\ell\|_2\le 1$, and already for linear activations. Thus, a mechanism widely used to stabilize training can create severe worst-case instabilities that are invisible locally, i.e., from layerwise norm bounds alone. Parameter-Lipschitz explosion turns Lipschitz-dependent nonsmooth optimization guarantees into exponential-in-dimension bounds for deep normalized networks, while input-Lipschitz explosion makes worst-case Lipschitz-based generalization and robustness certificates vacuous. The latter also yields a concrete rank-separation mechanism for adversarial vulnerability. The theory predicts that perturbations introducing new singular directions should be amplified much more strongly than perturbations at the same noise level that remain within the input's original singular subspace. Experiments on MNIST, Fashion-MNIST, and CIFAR-10 support this prediction, showing that rank-creating perturbations cause substantially sharper drops in accuracy, confidence, and margins than same-subspace perturbations. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Machine Learning (stat.ML). Pagina del lavoro: https://arxiv.org/abs/2611.08775 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-09T22:30:47.307Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).