Metraipolis › archive › dc_9fe73ab5a129

Manifold Projection and Iterative Autoencoder Refinement for Masked Language Modeling — arXiv:2609.30288

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.30288
License (as archived)
CC0-1.0
Archived
2026-10-08T02:03:52.934Z
SHA-256 of the archived text
3bcd2fdc795082a1598396e4ce909fceea36134d541b5d8a2e38e3e26ff38b39
Stable URL of this page
https://metraipolis.com/documento/dc_9fe73ab5a129

How to cite

Manifold Projection and Iterative Autoencoder Refinement for Masked Language Modeling — arXiv:2609.30288 — arXiv (en), https://arxiv.org/abs/2609.30288 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9fe73ab5a129 (Metraipolis archive dc_9fe73ab5a129, sha256 3bcd2fdc795082a1598396e4ce909fceea36134d541b5d8a2e38e3e26ff38b39)
BibTeX
@misc{dc9fe73ab5a129,
  title = {Manifold Projection and Iterative Autoencoder Refinement for Masked Language Modeling — arXiv:2609.30288},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.30288},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9fe73ab5a129; sha256 3bcd2fdc795082a1598396e4ce909fceea36134d541b5d8a2e38e3e26ff38b39},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.30288. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In Transformer-based masked language models, attention is the primary mechanism for context mixing, but there are other ways to mix data across tokens. Recent attention-free mixers replace attention with fixed or hypernetwork-generated MLPs, alternating their dynamic, content-dependent weighting for computational simplicity. We build an alternative that gets the same property from a low-rank bottleneck autoencoder. We replace attention with a stack of autoencoder-based mixing modules, one operating over local neighborhoods, one over the full sequence, and one across attention heads, each compressing and reconstructing its input through a bottleneck, and its width is a hyperparameter rather than a training effect. In masked positions, we introduce an iterative refinement procedure that has two distinct steps. A pulling step that pulls an embedding representation toward a weighted average of its neighbors, and a correcting step that projects the result back to the learned manifold via an autoencoder. Our architecture achieves a significant portion of attention's performance at about $1.9 \times$ fewer FLOPs when pretrained on C4 and evaluated with parameter-matched BERT baselines. Our model equals parameter-matched BERT and TinyBERT baselines on the rarest-token frequency bucket using a frequency-aware training schedule that samples rare tokens more than uniformly for the masking tasks.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-28 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.30288 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T01:38:49.417Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).