Metraipolis › archive › dc_05686b0bca01

Where to Connect? Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers — arXiv:2611.06844

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.06844
License (as archived)
CC0-1.0
Archived
2026-11-05T18:30:13.759Z
SHA-256 of the archived text
412d0e383ff44b76e440dd7198b542a8c0a4fd5d770d5fda7ba9a18e2c02e29e
Stable URL of this page
https://metraipolis.com/documento/dc_05686b0bca01

How to cite

Where to Connect? Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers — arXiv:2611.06844 — arXiv (en), https://arxiv.org/abs/2611.06844 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_05686b0bca01 (Metraipolis archive dc_05686b0bca01, sha256 412d0e383ff44b76e440dd7198b542a8c0a4fd5d770d5fda7ba9a18e2c02e29e)
BibTeX
@misc{dc05686b0bca01,
  title = {Where to Connect? Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers — arXiv:2611.06844},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.06844},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_05686b0bca01; sha256 412d0e383ff44b76e440dd7198b542a8c0a4fd5d770d5fda7ba9a18e2c02e29e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.06844. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A central yet under-explored question in Multimodal Large Language Models (MLLMs) is where to connect---how visual information from a hierarchical vision encoder should be wired into the layer-wise semantics of a large language model. Mainstream MLLMs inject a fixed single-layer visual representation into the LLM and treat all LLM layers as a single consumer, ignoring that different layers demand different visual granularities. Recent multi-layer fusion methods either compress hierarchical features on the encoder side or rely on predefined sparse or hierarchical layer-to-layer connections, and therefore stop short of modeling adaptive cross-layer interactions between the ViT and LLM hierarchies. We address this gap with DGP (Dynamic Gated Pathways), a routing mechanism that establishes all-to-all connections between every ViT layer and every LLM layer, with each LLM layer dynamically gating its preference over visual layers conditioned on its own semantic state. Built on top of LLaVA-1.5-7B and 13B, DGP delivers consistent gains over the LLaVA-1.5-7B baseline (e.g., +4.2 on SQA$^{I}$, +3.3 on MM-Vet, +3.1 on VizWiz and LLaVA$^{W}$) and surpasses prior multi-layer visual fusion methods on most benchmarks. Layer-wise routing analyses further show that different LLM layers do select different visual granularities on demand, supporting both the effectiveness and interpretability of the proposed dynamic gated pathways. The code is available: this https URL .

Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.CV, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2611.06844 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:19:49.066Z dalla pagina https://arxiv.org/list/cs.CV/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).