Metraipolis › archive › dc_7d89aa9dbfef

How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining — arXiv:2609.35457

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.35457
License (as archived): CC0-1.0
Archived: 2026-10-07T23:54:48.077Z
SHA-256 of the archived text: 264b963431c9c921c7fa67b62a31d61134884adb65e29f4f8dc01ccdc1e73b00
Stable URL of this page: https://metraipolis.com/documento/dc_7d89aa9dbfef

How to cite

How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining — arXiv:2609.35457 — arXiv (en), https://arxiv.org/abs/2609.35457 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7d89aa9dbfef (Metraipolis archive dc_7d89aa9dbfef, sha256 264b963431c9c921c7fa67b62a31d61134884adb65e29f4f8dc01ccdc1e73b00)

BibTeX

@misc{dc7d89aa9dbfef,
  title = {How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining — arXiv:2609.35457},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35457},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7d89aa9dbfef; sha256 264b963431c9c921c7fa67b62a31d61134884adb65e29f4f8dc01ccdc1e73b00},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35457. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Most modern multimodal large language models (MLLMs) build on a pretrained visual encoder that provides a strong visual prior. Encoder-free MLLMs instead learn visual representations directly from raw pixels, offering a simple and unified architecture, but their scaling behavior has not been systematically characterized. To fill this gap, we compare scaling laws for encoder-free and encoder-based MLLMs and report three main findings: (1) Removing the visual encoder shifts the compute-optimal allocation for the multimodal objective toward larger models, while leaving that for text nearly unchanged. (2) The two architectures exhibit nearly overlapping loss--compute frontiers on the text objective, but diverge on the multimodal objective: encoder-free models underperform at small scales yet are predicted to catch up at around $10^{22}$ FLOPs, well within practical pretraining budgets. (3) Without a visual encoder, the language model learns to take over its role via vision-specific adaptation: bidirectional interactions among visual tokens become increasingly beneficial as training compute grows, visual processing shifts toward earlier layers, and expert routing for visual tokens becomes more concentrated. Overall, our results indicate that the advantage of the visual prior provided by a pretrained encoder diminishes with scale, positioning encoder-free architectures as a promising direction for multimodal pretraining.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CV).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2609.35457 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T23:29:27.574Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).