Metraipolis › archive › dc_3bfa94e559cc
Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents — arXiv:2610.41432
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.41432
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T07:35:13.968Z
- SHA-256 of the archived text
bdc7671569404e3a459ce2aeaa7f720fc33eb198da7c056491c832afeac83411- Stable URL of this page
https://metraipolis.com/documento/dc_3bfa94e559cc
How to cite
Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents — arXiv:2610.41432 — arXiv (en), https://arxiv.org/abs/2610.41432 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3bfa94e559cc (Metraipolis archive dc_3bfa94e559cc, sha256 bdc7671569404e3a459ce2aeaa7f720fc33eb198da7c056491c832afeac83411)
BibTeX
@misc{dc3bfa94e559cc,
title = {Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents — arXiv:2610.41432},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.41432},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3bfa94e559cc; sha256 bdc7671569404e3a459ce2aeaa7f720fc33eb198da7c056491c832afeac83411},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41432. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Computer-use agents increasingly tackle tasks ranging from everyday desktop activities to long-horizon professional workflows. Yet interfaces centered on screenshots and mouse-and-keyboard actions can impose a human-interaction prior, requiring repeated screen interpretation while leaving the computer's semantic and programmatic capabilities underexposed. We argue for exposing these capabilities and letting models choose and combine them. We introduce Hybrid Computer Use (HCU), a unified interface for general agents that combines three complementary channels: programmatic interaction through code and command-line tools, semantic interaction through accessibility (AX), and visual interaction through graphical user interfaces (GUI). Our HCU supports efficient AX interaction and guards against stale-state actions during channel switching. A supplementary HCU skill guides agents in selecting and coordinating channels to use the unified interface effectively. Our designed AX tools return compact observations, execute actions in batches, and report their effects as AX diffs, so the model needs fewer round trips. On OSWorld 2.0, HCU achieves the highest Partial and Binary scores among the compared systems. Analyzing agent trajectories, we find that all evaluated models improve with HCU yet allocate work across the channels differently and consistently outperform compared baselines in terms of both efficiency and task success, highlighting the value of unified computer-use interfaces that allow flexible channel selection and combination. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Human-Computer Interaction (cs.HC) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.41432 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T07:29:48.001Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).