Metraipolis › archive › dc_e02ff3d4b196

An Information Budget for Label-Noise Memorization, and What Statistical Complexity Tracks: A Study with Multilayer Perceptrons — arXiv:2611.04199

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04199
License (as archived)
CC0-1.0
Archived
2026-11-04T12:40:47.045Z
SHA-256 of the archived text
5e59cc372f7dfdfb9093272ebcda87140a121c245c0b39ee2f75403b8f3e350a
Stable URL of this page
https://metraipolis.com/documento/dc_e02ff3d4b196

How to cite

An Information Budget for Label-Noise Memorization, and What Statistical Complexity Tracks: A Study with Multilayer Perceptrons — arXiv:2611.04199 — arXiv (en), https://arxiv.org/abs/2611.04199 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e02ff3d4b196 (Metraipolis archive dc_e02ff3d4b196, sha256 5e59cc372f7dfdfb9093272ebcda87140a121c245c0b39ee2f75403b8f3e350a)
BibTeX
@misc{dce02ff3d4b196,
  title = {An Information Budget for Label-Noise Memorization, and What Statistical Complexity Tracks: A Study with Multilayer Perceptrons — arXiv:2611.04199},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04199},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e02ff3d4b196; sha256 5e59cc372f7dfdfb9093272ebcda87140a121c245c0b39ee2f75403b8f3e350a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04199. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

How much label noise has a network memorized, and when? We study a multilayer perceptron and what statistical complexity adds to a description of its predictions. Standard identities decompose predictive information into signal, memorization, and heterogeneity. The memorization term vanishes in the population under the stated noise-independence condition. A Fano-type bound relates it to training accuracies, but the memorized fraction M provides no upper bound. Across 89 MNIST runs with ten pre-specified hypotheses (five supported, four unsupported, one not testable), normalized memorization information rose with M on training samples, remained near zero on held-out samples with fresh noisy labels, and was close to its Fano bound. Unlike raw M, it has a zero population null under the independence condition and chance correction largely removed this difference. When every label was replaced by an incorrect class, test accuracy fell to 0.5%, yet predictions retained information about the true class. Mean statistical complexity peaked before memorization at corruption rates up to 40%. At higher rates, its peak fell within the memorization phase, near the epoch when mean training confidence reached 0.6 and the two could not be separated. As a stopping rule, it matched established rules only at low noise and detected corrupted samples less well than simple label-free scores. An exploratory extension of 280 runs covering Fashion-MNIST, pair-flip noise, and four training variants yielded similar results under uniform noise, but not pair-flip noise, where complexity peaked within two epochs. Conclusions are limited to this architecture, two small datasets, and synthetic noise.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.04199 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T12:29:47.285Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).