Metraipolis › archive › dc_ca8fd014d1ce

Catastrophic Overfitting is Weight Drift the Training Loss Cannot See — arXiv:2611.06082

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.06082
License (as archived)
CC0-1.0
Archived
2026-11-05T18:21:12.103Z
SHA-256 of the archived text
00ff35fed4ebb00ff667400b210484ace6fef7594c915a8f8e5ddb3cf20e1044
Stable URL of this page
https://metraipolis.com/documento/dc_ca8fd014d1ce

How to cite

Catastrophic Overfitting is Weight Drift the Training Loss Cannot See — arXiv:2611.06082 — arXiv (en), https://arxiv.org/abs/2611.06082 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ca8fd014d1ce (Metraipolis archive dc_ca8fd014d1ce, sha256 00ff35fed4ebb00ff667400b210484ace6fef7594c915a8f8e5ddb3cf20e1044)
BibTeX
@misc{dcca8fd014d1ce,
  title = {Catastrophic Overfitting is Weight Drift the Training Loss Cannot See — arXiv:2611.06082},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.06082},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ca8fd014d1ce; sha256 00ff35fed4ebb00ff667400b210484ace6fef7594c915a8f8e5ddb3cf20e1044},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.06082. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Fast adversarial training (FAT) makes robust training affordable by replacing the multi-step attack with a single-step one, but it suffers from catastrophic overfitting (CO): partway through training, robustness to multi-step attacks collapses to near zero while the single-step attack used for training keeps being repelled. We show that CO is a quiet escape of the weights. Across the event the network's clean behavior quietly improves, and moving from the weights just before CO to those just after it changes the training objective only marginally while destroying most of the robustness. The training objective is therefore nearly blind to the displacement it causes, so it exerts no force to undo it. The exponential moving average (EMA) of the weights, which FAT protocols already maintain, does not accumulate this drift and stays robust through the entire event. This observation suggests a nearly free fix: when validation robustness drops, reset the weights to their EMA. We build this guard on top of a coupling-based base method designed so that its average stays inside the robust region. The resulting training removes CO in every one of our runs and attains the strongest AutoAttack accuracy among fast methods on all three benchmark datasets, including a clear margin on CIFAR-100, where it also clears the multi-step reference at roughly half its cost. Code is available at this https URL .

Tema degli esploratori: fiducia digitale e crittografia (elenco arXiv cs.CR, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Cryptography and Security (cs.CR).
Pagina del lavoro: https://arxiv.org/abs/2611.06082 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:15:18.893Z dalla pagina https://arxiv.org/list/cs.CR/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).