Metraipolis › archive › dc_5bd5ce8dca43
OSWorld-Pro: Process-based Evaluation for Computer Use Agents — arXiv:2609.24890
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.24890
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T06:50:15.781Z
- SHA-256 of the archived text
cabb190690569e846ec78bf8ffead90677f59c8b3a178e4eedbc84dd33ccd2c4- Stable URL of this page
https://metraipolis.com/documento/dc_5bd5ce8dca43
How to cite
OSWorld-Pro: Process-based Evaluation for Computer Use Agents — arXiv:2609.24890 — arXiv (en), https://arxiv.org/abs/2609.24890 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5bd5ce8dca43 (Metraipolis archive dc_5bd5ce8dca43, sha256 cabb190690569e846ec78bf8ffead90677f59c8b3a178e4eedbc84dd33ccd2c4)
BibTeX
@misc{dc5bd5ce8dca43,
title = {OSWorld-Pro: Process-based Evaluation for Computer Use Agents — arXiv:2609.24890},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.24890},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5bd5ce8dca43; sha256 cabb190690569e846ec78bf8ffead90677f59c8b3a178e4eedbc84dd33ccd2c4},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.24890. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Evaluation of Computer-Use Agents (CUAs) is often limited to the final deliverables they create (at the end of hundreds of steps) and assessed with functional verifiers, as seen in OSWorld. However, such evaluation of end-state performance lacks transparency into how and why agents fail in various tasks, obfuscating critical insight for subsequent improvement. For instance, agents that err during keyboard inputs would require a different mitigation strategy from those that fail to precisely provide click-based inputs on the graphical UI. We introduce OSWorld-Pro: a set of over 300 tasks containing over 2800 subgoals to enable the procedural evaluation of CUAs grounded in over 67,000 human annotations. We use robust human-aligned LLM-Judges to evaluate the fulfillment of OSWorld-Pro subgoals and thereby reveal the progress that models make throughout a series of sequentially dependent subgoals. Our findings reveal that OSWorld-Pro is challenging even for state-of-the-art LLMs, with top performers like Claude Opus 5 achieving only 75.7% vs. 83.4% on OSWorld. Furthermore, we identify critical process-focused failure modes of various models (e.g. subgoal-irrelevant actions and click-based mistakes) to provide insights to improve performance and efficiency of CUAs. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.24890 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T06:48:45.068Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).