Metraipolis › archive › dc_3fe9bc1b67bb
Computer-Use Agents Gain Little Benefit from a World Model When Conditioned on Tutorials — arXiv:2611.04073
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.04073
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-04T18:53:58.386Z
- SHA-256 of the archived text
2d20da6b8407c2496a9648a31de66b896ec6ed3fb8bffd944d5fe9dda8f83cf8- Stable URL of this page
https://metraipolis.com/documento/dc_3fe9bc1b67bb
How to cite
Computer-Use Agents Gain Little Benefit from a World Model When Conditioned on Tutorials — arXiv:2611.04073 — arXiv (en), https://arxiv.org/abs/2611.04073 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3fe9bc1b67bb (Metraipolis archive dc_3fe9bc1b67bb, sha256 2d20da6b8407c2496a9648a31de66b896ec6ed3fb8bffd944d5fe9dda8f83cf8)
BibTeX
@misc{dc3fe9bc1b67bb,
title = {Computer-Use Agents Gain Little Benefit from a World Model When Conditioned on Tutorials — arXiv:2611.04073},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.04073},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3fe9bc1b67bb; sha256 2d20da6b8407c2496a9648a31de66b896ec6ed3fb8bffd944d5fe9dda8f83cf8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04073. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Recent work has improved computer-use agents (CUAs) by combining retrieved task-relevant tutorials and look-ahead planning using world models with MLLM judges. However, the contribution of each component -- tutorials, the judge, and the world model -- to overall improvement has not been evaluated in isolation. To fill this gap, we first construct OSWorld-Tutorial and ScienceBoard-Tutorial: two datasets for CUAs with curated multimodal tutorials, covering everyday and scientific domains. We then propose RAG-PolicyJudge (RAG-PJ), a policy-judge framework for CUAs conditioned on task-specific tutorials: the policy proposes candidate actions, and the judge selects among them. To isolate the effect of world models given tutorials, we compare RAG-PJ with and without common world models: using either zero-shot text-based descriptions of next states or an oracle that returns ground-truth visual observations. We find that RAG-PJ gains little benefit from these rollouts when tutorials are present, even with perfect visual futures. This suggests prior world-model gains in tutorial-conditioned methods come mainly from tutorial understanding and the judge's ability to select the proposed actions, rather than from predicted rollouts. Error analysis reveals that the bottleneck is not rollout fidelity, but the judge's ability to reason over future states and recognize task completion. This highlights the need to improve not just next-state prediction in world models, but also reasoning over their predicted rollouts. Datasets and code are available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.04073 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-04T18:35:37.422Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).