Metraipolis › archive › dc_862e730e37c0
A Theoretical Framework for Masked Pretraining (MPT) — arXiv:2609.06460
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.06460
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T13:56:46.141Z
- SHA-256 of the archived text
a2fdff05b9babbe91da3603ec39ecf2e916ee6cc0edb07409ce70a3f9aab2972- Stable URL of this page
https://metraipolis.com/documento/dc_862e730e37c0
How to cite
A Theoretical Framework for Masked Pretraining (MPT) — arXiv:2609.06460 — arXiv (en), https://arxiv.org/abs/2609.06460 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_862e730e37c0 (Metraipolis archive dc_862e730e37c0, sha256 a2fdff05b9babbe91da3603ec39ecf2e916ee6cc0edb07409ce70a3f9aab2972)
BibTeX
@misc{dc862e730e37c0,
title = {A Theoretical Framework for Masked Pretraining (MPT) — arXiv:2609.06460},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.06460},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_862e730e37c0; sha256 a2fdff05b9babbe91da3603ec39ecf2e916ee6cc0edb07409ce70a3f9aab2972},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.06460. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Recently, Masked Pretraining (MPT) based on reconstruction pretraining tasks has risen to a promising self-supervised learning paradigm across various domains and achieves remarkable performance in multiple downstream tasks. However, the theoretical understanding of the working mechanism behind MPT is still limited. In this paper, we introduce a new theoretical framework to analyze MPT and understand the crucial role of masking in extracting meaningful representations. We establish theoretical connections between MPT and another popular self-supervised paradigm: contrastive learning. We prove that the masking technique implicitly creates positive pairs that are semantically similar and the reconstruction loss pulls them together in the feature space. Besides, as a result of the implicit alignment, we point out the dimensional collapse issue of MPT and propose a Uniformity-enhanced MPT (U-MPT) loss that can effectively address this issue and bring significant improvements in downstream tasks including linear evaluation, cross-dataset fine-tuning and out-of-distribution generalization on real-world data sets. Furthermore, we establish downstream guarantees of U-MPT and theoretically analyze the influence of masking strategies. Based on the theoretical analysis, we propose a new masking strategy which enhances the downstream performance of MPT and explains current improvements of masking strategies with our theoretical perspective. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.06460 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T13:39:42.872Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).