Metraipolis › archive › dc_fe187a007223

AutoGUIWorld: Image Generators as Visual World Models for GUI Agent — arXiv:2610.01215

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.01215
License (as archived)
CC0-1.0
Archived
2026-10-06T19:13:03.714Z
SHA-256 of the archived text
cc135d7c3626988b4c4396d34faabffe7ab1e4f54e20aa9dfdaa6ac5bbe07599
Stable URL of this page
https://metraipolis.com/documento/dc_fe187a007223

How to cite

AutoGUIWorld: Image Generators as Visual World Models for GUI Agent — arXiv:2610.01215 — arXiv (en), https://arxiv.org/abs/2610.01215 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fe187a007223 (Metraipolis archive dc_fe187a007223, sha256 cc135d7c3626988b4c4396d34faabffe7ab1e4f54e20aa9dfdaa6ac5bbe07599)
BibTeX
@misc{dcfe187a007223,
  title = {AutoGUIWorld: Image Generators as Visual World Models for GUI Agent — arXiv:2610.01215},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.01215},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fe187a007223; sha256 cc135d7c3626988b4c4396d34faabffe7ab1e4f54e20aa9dfdaa6ac5bbe07599},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01215. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

GUI agents require high-quality interaction trajectories to learn how software environments respond to actions, maintain state, and support multi-step workflows. However, the diversity of available trajectories is constrained by the applications, interface states, and workflows accessible in the underlying environments. Expanding this coverage requires deploying increasingly diverse and complex software, with specialized applications imposing additional installation, configuration, and runtime costs. We introduce AutoGUIWorld, a data generation framework that combines the visual priors of image generators with the task knowledge of a planner to synthesize GUI interaction trajectories without deploying or running the corresponding software environments. AutoGUIWorld samples initial GUI scenes from structured specifications of operating-system context, visual appearance, and interface state, and generates tasks conditioned on those scenes. A planner then specifies atomic actions and their intended visual consequences, while an image generator iteratively edits the current screenshot to produce subsequent observations. Action grounding and transition-level quality filtering yield 79,266 spatially annotated step-level training samples across Ubuntu, Windows, macOS, and Chrome. Fine-tuning Qwen3.5-35B-A3B on AutoGUIWorld trajectories improves the mean task score on OSWorld from 33.0% to 40.8% and the task success rate on ScienceBoard from 14.0% to 32.2%. These results show that generated trajectories improve GUI-agent performance on real desktop and scientific tasks.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-10-02 in cs.CV).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2610.01215 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T17:37:36.763Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).