Metraipolis › archive › dc_dd448da45ac4
The hidden advantage of mask resampling: a theory of masked autoencoders — arXiv:2610.01578
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.01578
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T15:33:02.700Z
- SHA-256 of the archived text
4c740146b5ebc6cbd76baf4fd0e1efe09a4b928bdce68d996da294def7df440e- Stable URL of this page
https://metraipolis.com/documento/dc_dd448da45ac4
How to cite
The hidden advantage of mask resampling: a theory of masked autoencoders — arXiv:2610.01578 — arXiv (en), https://arxiv.org/abs/2610.01578 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dd448da45ac4 (Metraipolis archive dc_dd448da45ac4, sha256 4c740146b5ebc6cbd76baf4fd0e1efe09a4b928bdce68d996da294def7df440e)
BibTeX
@misc{dcdd448da45ac4,
title = {The hidden advantage of mask resampling: a theory of masked autoencoders — arXiv:2610.01578},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.01578},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dd448da45ac4; sha256 4c740146b5ebc6cbd76baf4fd0e1efe09a4b928bdce68d996da294def7df440e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01578. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Why can masked prediction learn useful representations that unmasked reconstruction misses? We study this question in a high-dimensional model of a masked autoencoder (MAE) trained on data with shared latent structure and heterogeneous noise. We prove that masked linear reconstruction can recover the latent feature at linear sample complexity in regimes where unmasked linear reconstruction, equivalent to PCA, fails. The analysis also quantifies the statistical advantage of mask resampling, an established ingredient of masked pretraining. By introducing a fixed collection of $K$ masks per sample, we characterize its effect on feature recovery and downstream performance, identifying regimes where greater mask diversity lowers sample complexity. Guided by this prediction, we find that random cropping and flipping in standard image-training pipelines can obscure the advantage of mask resampling by renewing the prediction task even when the patch mask is fixed. Removing these transformations reveals a downstream advantage for dynamic over static masking in CNN autoencoders and vision transformers. A complementary BERT pilot finds benefits from greater mask diversity on downstream language tasks. Our results separate the benefit of the masked prediction objective from that of mask diversity, and show how a tractable theory can guide experiments that uncover advantages hidden by standard training practices. Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-10-02 in stat.ML). Categorie arXiv: Machine Learning (stat.ML) ; Disordered Systems and Neural Networks (cond-mat.dis-nn); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.01578 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T15:12:55.876Z dalla pagina https://arxiv.org/catchup/stat.ML/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).