Metraipolis › archive › dc_3e07b070e06d

Training-Free Speech-Centric Omni Understanding with Frozen VLMs — arXiv:2609.04242

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.04242
License (as archived)
CC0-1.0
Archived
2026-10-16T23:41:27.471Z
SHA-256 of the archived text
d7deee745d1f966355ca6e9b2284f957dd06ff36239599c322668c3e1703e4ae
Stable URL of this page
https://metraipolis.com/documento/dc_3e07b070e06d

How to cite

Training-Free Speech-Centric Omni Understanding with Frozen VLMs — arXiv:2609.04242 — arXiv (en), https://arxiv.org/abs/2609.04242 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3e07b070e06d (Metraipolis archive dc_3e07b070e06d, sha256 d7deee745d1f966355ca6e9b2284f957dd06ff36239599c322668c3e1703e4ae)
BibTeX
@misc{dc3e07b070e06d,
  title = {Training-Free Speech-Centric Omni Understanding with Frozen VLMs — arXiv:2609.04242},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.04242},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3e07b070e06d; sha256 d7deee745d1f966355ca6e9b2284f957dd06ff36239599c322668c3e1703e4ae},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04242. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Audio-visual understanding remains challenging because models must jointly interpret spoken content, visual events, and their temporal relationships. Existing omni models typically introduce dedicated audio encoders and rely on expensive audio-video-text training, tightly coupling omni capability to specific VLM backbones and potentially weakening their existing visual and reasoning abilities. This raises three questions: whether native omni training is necessary for every new VLM, whether speech-centric omni capability can be added while preserving the original backbone, and where richer acoustic representations remain essential. We introduce Training-Free Omni (TFO), a plug-and-play framework that converts a frozen VLM into a speech-centric omni model without architectural modification, or multimodal re-alignment. TFO uses Whisper to extract confidence-filtered, timestamped transcripts and routes them through the VLM's existing language interface, while leaving its visual pathway unchanged. Across matched comparisons with native omni models on 56 benchmarks and 21 languages, TFO is competitive on audio-visual understanding, improves average audio-only performance across all five model settings, and achieves substantial multilingual speech gains. Freezing the VLM also generally preserves stronger image/video understanding, visual grounding, coding, mathematical reasoning, and medical question answering than the corresponding native omni checkpoints. These results show that strong speech-centric omni understanding can often be obtained through modular audio-to-language routing rather than costly backbone-specific training.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-07 in cs.CV).
Categorie arXiv: Audio and Speech Processing (eess.AS) ; Computer Vision and Pattern Recognition (cs.CV); Sound (cs.SD).
Pagina del lavoro: https://arxiv.org/abs/2609.04242 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T23:16:11.468Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).