Metraipolis › archive › dc_5c5f02f7ccc8

OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs — arXiv:2608.21360

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.21360
License (as archived)
CC0-1.0
Archived
2026-10-18T10:56:25.523Z
SHA-256 of the archived text
f513c8c6f0aa7da2c92ddf0a720cfde84d3c8f6cdd784bab50dd1e7d7819adfc
Stable URL of this page
https://metraipolis.com/documento/dc_5c5f02f7ccc8

How to cite

OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs — arXiv:2608.21360 — arXiv (en), https://arxiv.org/abs/2608.21360 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5c5f02f7ccc8 (Metraipolis archive dc_5c5f02f7ccc8, sha256 f513c8c6f0aa7da2c92ddf0a720cfde84d3c8f6cdd784bab50dd1e7d7819adfc)
BibTeX
@misc{dc5c5f02f7ccc8,
  title = {OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs — arXiv:2608.21360},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.21360},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5c5f02f7ccc8; sha256 f513c8c6f0aa7da2c92ddf0a720cfde84d3c8f6cdd784bab50dd1e7d7819adfc},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.21360. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recent omni-modal large language models (Omni-LLMs) show great potential as real-time video assistants, which continuously perceive environments and guide users to achieve specific goals. Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help. Evaluating this is rather challenging, as the model's unpredictable response dynamically changes the user's subsequent actions, which static offline datasets cannot accommodate. To address this bottleneck, we introduce OmniAssistBench. To solve the issue of diverging interaction paths where the same user goal can be achieved through various methods, we provide models with predefined priors derived from the source video, requiring them to guide users along the exact same routes. Since real interaction videos are rare, we construct the dataset by reverse-engineering existing Internet videos. We deduce logical user goals and segment the videos into multi-turn clips to simulate continuous interactions. This rigorous pipeline required over 1000 expert person-hours to build the dataset. Results show that the proprietary Gemini-3-Pro reaches 66.4 out of the max point of 100, while the open-source Qwen3-Omni-Instruct achieves 51.2. Although current models generally understand user inputs, they frequently provide incorrect or incomplete answers. Specifically, they struggle with visual prompts (e.g., hand gestures), fail to maintain historical context during multi-turn interactions, and fail to delay response until the target event. Results indicate substantial room for improvement before models can become reliable assistants.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-08-24 in cs.CV).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2608.21360 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T10:48:13.611Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-08-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).