Metraipolis › archive › dc_3dc130aec6bf

Adaptation of Computer Use Agents to New Software with User Manuals — arXiv:2610.23012

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.23012
License (as archived): CC0-1.0
Archived: 2026-10-19T08:42:23.065Z
SHA-256 of the archived text: a0f378c4dce39053271488694be4422f88cecf72ebdbb1f308aa8d7f3105eb17
Stable URL of this page: https://metraipolis.com/documento/dc_3dc130aec6bf

How to cite

Adaptation of Computer Use Agents to New Software with User Manuals — arXiv:2610.23012 — arXiv (en), https://arxiv.org/abs/2610.23012 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3dc130aec6bf (Metraipolis archive dc_3dc130aec6bf, sha256 a0f378c4dce39053271488694be4422f88cecf72ebdbb1f308aa8d7f3105eb17)

BibTeX

@misc{dc3dc130aec6bf,
  title = {Adaptation of Computer Use Agents to New Software with User Manuals — arXiv:2610.23012},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.23012},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3dc130aec6bf; sha256 a0f378c4dce39053271488694be4422f88cecf72ebdbb1f308aa8d7f3105eb17},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.23012. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recent advances in multimodal foundation models have significantly improved the ability to operate real-world software through graphical user interfaces (GUIs) using visual observations and natural language instructions. However, maintaining strong performance on unseen applications remains challenging, as practical tasks often require long-horizon procedural knowledge and application-specific operation patterns. In addition, constructing executable tasks and collecting corresponding trajectories for training still heavily rely on human-annotated data, incurring substantial cost. In this paper, we propose MAGE (Manual-augmented computer use AGent Execution and learning), a framework that leverages user manuals as a scalable source of procedural knowledge to improve both inference and training. During execution, MAGE retrieves task-relevant manual content and uses a planner to generate explicit procedural guidance for CUAs. To further account for the discrepancies between manuals and actual GUI environments, such as version differences or user settings, MAGE also adapts the agent policy through environment interaction. During training, MAGE converts manuals into executable task-plan pairs and uses the plans to guide environment rollouts toward task-relevant states; the resulting trajectories are then used to optimize the policy without human-authored demonstrations. This unified design enables effective utilization of manuals for both task execution and training, reducing reliance on human-authored training data. Experimental results across six applications demonstrate improvements in task success rates over strong baselines, highlighting the effectiveness of manual-guided planning and policy adaptation in new software environments. We provide the code and reproduction scripts at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.23012 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:39:10.052Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).