Metraipolis › archive › dc_959216a708ca
Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds — arXiv:2611.08286
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.08286
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-06T14:53:44.847Z
- SHA-256 of the archived text
687e4bb73dba203907e2360713a563de18c20f8565b2c5d7282c5b506bfb1b2b- Stable URL of this page
https://metraipolis.com/documento/dc_959216a708ca
How to cite
Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds — arXiv:2611.08286 — arXiv (en), https://arxiv.org/abs/2611.08286 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_959216a708ca (Metraipolis archive dc_959216a708ca, sha256 687e4bb73dba203907e2360713a563de18c20f8565b2c5d7282c5b506bfb1b2b)
BibTeX
@misc{dc959216a708ca,
title = {Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds — arXiv:2611.08286},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.08286},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_959216a708ca; sha256 687e4bb73dba203907e2360713a563de18c20f8565b2c5d7282c5b506bfb1b2b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.08286. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Visual interaction models that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users. We contribute three simulators that recreate everyday tasks and human-aligned action spaces across cooking, household manipulation, and mobile-interface domains, together with a persona-conditioned user simulation harness that collaborates with the assistant while incorporating controlled execution errors and other out-of-plan behaviors. We instantiate the framework as IVG-Bench, a benchmark with 315 interaction scenarios. Evaluating state-of-the-art VLMs as visual interaction models reveals substantial limitations: the best-performing model, Gemini 4 Argon, completes only 53.4% of tasks within the time limit, exhibits significant weaknesses in factual grounding and actionable guidance, and is often too slow to prevent errors when response latency is considered. We verify our evaluation protocol with a human study which finds 78.5% agreement between our interaction-quality metric and participant preferences. We further show that the environments provide useful learning signals: in a proof-of-concept GRPO experiment, closed-loop training improves both task success and interaction quality for Qwen3.5-4B model in our cooking environment. Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.CV, nuovi lavori del giorno). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV). Pagina del lavoro: https://arxiv.org/abs/2611.08286 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-06T14:25:42.628Z dalla pagina https://arxiv.org/list/cs.CV/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).