Metraipolis › archive › dc_1144f37d5366

ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents — arXiv:2608.18307

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.18307
License (as archived)
CC0-1.0
Archived
2026-10-18T13:44:27.162Z
SHA-256 of the archived text
712acce11eaa6494e6571b7b227937cf3bce7c76f4bcb204d9e1da2112099307
Stable URL of this page
https://metraipolis.com/documento/dc_1144f37d5366

How to cite

ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents — arXiv:2608.18307 — arXiv (en), https://arxiv.org/abs/2608.18307 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1144f37d5366 (Metraipolis archive dc_1144f37d5366, sha256 712acce11eaa6494e6571b7b227937cf3bce7c76f4bcb204d9e1da2112099307)
BibTeX
@misc{dc1144f37d5366,
  title = {ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents — arXiv:2608.18307},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.18307},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1144f37d5366; sha256 712acce11eaa6494e6571b7b227937cf3bce7c76f4bcb204d9e1da2112099307},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.18307. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Current evaluation of computer-use agents is split between long-horizon workflow benchmarks and atomic GUI-grounding tests. This leaves an under-instrumented middle layer: realistic component-centered interactions (e.g., toggle a button set) that are short enough to diagnose and rich enough to capture the burdens of modern interfaces. We present ComponentBench, a benchmark and diagnostic pipeline for component-level evaluation of computer-use agents on modern web UIs. ComponentBench is organized around a library-agnostic ontology of 97 canonical UI components instantiated as 2,910 programmatically verified tasks across widely used component libraries, paired with cleaned human reference trajectories that enable evaluation of both task success and interaction efficiency. Beyond task collection, we introduce a scalable pipeline for auditing realized structural difficulty after implementation and synthesizing structured failure analyses across tasks and component families. Evaluating seven models -- GPT-5.4, Gemini 3 Flash, GPT-5.4 mini, GPT-5 mini, Gemini 3.1 Flash-Lite, Qwen3-VL-235B, and UI-TARS-1.5-7B -- across four observation and action spaces, we show that these design choices critically impact performance. Within a single shared harness, changing only the observation and action space shifts task success by more than 30% for the same model: GPT-5 mini falls from 83.1% with accessibility-tree observations to 48.9% with coordinate-only Pixel control. Moreover, even the fastest configuration takes 3.7x as long as the matched human reference, and spatial manipulations that are trivial for humans continue to challenge current agents.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-20 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Human-Computer Interaction (cs.HC).
Pagina del lavoro: https://arxiv.org/abs/2608.18307 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T13:41:17.857Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).