Metraipolis › archive › dc_41597bf6b725

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue — arXiv:2609.21465

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.21465
License (as archived)
CC0-1.0
Archived
2026-10-10T01:08:31.685Z
SHA-256 of the archived text
40430a4f8aa0193158e44e9003d3e275ef9a397395cce6af1a289e7cdecd119f
Stable URL of this page
https://metraipolis.com/documento/dc_41597bf6b725

How to cite

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue — arXiv:2609.21465 — arXiv (en), https://arxiv.org/abs/2609.21465 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_41597bf6b725 (Metraipolis archive dc_41597bf6b725, sha256 40430a4f8aa0193158e44e9003d3e275ef9a397395cce6af1a289e7cdecd119f)
BibTeX
@misc{dc41597bf6b725,
  title = {OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue — arXiv:2609.21465},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.21465},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_41597bf6b725; sha256 40430a4f8aa0193158e44e9003d3e275ef9a397395cce6af1a289e7cdecd119f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21465. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, good replies often depend on multimodal context and can be phrased in many ways, making keyword matching unreliable for evaluation. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. Replies are judged by a large language model based on explicit scoring criteria. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.AI).
Categorie arXiv: Audio and Speech Processing (eess.AS) ; Artificial Intelligence (cs.AI); Image and Video Processing (eess.IV).
Pagina del lavoro: https://arxiv.org/abs/2609.21465 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T00:23:42.539Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).