Metraipolis › archive › dc_42e19aa7fb5c
Benchmarking General Mobile Assistants in Challenging Real-World Scenarios — arXiv:2608.27477
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.27477
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T10:47:01.512Z
- SHA-256 of the archived text
0142a8cfa8fa1a1b32d6a0ffeb5b8bc8a338de82292d6ed57d7c9aa56b871591- Stable URL of this page
https://metraipolis.com/documento/dc_42e19aa7fb5c
How to cite
Benchmarking General Mobile Assistants in Challenging Real-World Scenarios — arXiv:2608.27477 — arXiv (en), https://arxiv.org/abs/2608.27477 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_42e19aa7fb5c (Metraipolis archive dc_42e19aa7fb5c, sha256 0142a8cfa8fa1a1b32d6a0ffeb5b8bc8a338de82292d6ed57d7c9aa56b871591)
BibTeX
@misc{dc42e19aa7fb5c,
title = {Benchmarking General Mobile Assistants in Challenging Real-World Scenarios — arXiv:2608.27477},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.27477},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_42e19aa7fb5c; sha256 0142a8cfa8fa1a1b32d6a0ffeb5b8bc8a338de82292d6ed57d7c9aa56b871591},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27477. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Graphical user interfaces have emerged as an important environment for evaluating autonomous AI agents on multimodal interactive tasks. Existing benchmarks such as AndroidWorld and MobileWorld provide strong foundations for mobile agent evaluation, but their application coverage and task design do not yet fully capture the diversity and complexity of realistic mobile use. We present GMA, a benchmark for evaluating general mobile assistants in challenging real-world scenarios. GMA introduces seven applications based on open-source projects, spanning domains such as lifestyle sharing and travel planning, and 300 tasks across four difficulty tiers, from atomic actions to complex multi-step workflows. We evaluate eight frontier models and find that performance declines substantially as task complexity increases, with current agents remaining far from reliably handling realistic user requirements. We further conduct controlled ablation studies of agent harness choices, including context retention and explicit state tracking, under a shared environment, model setting, and task taxonomy. Results show that appropriate harness design can meaningfully improve performance, particularly on demanding workflows, while the effectiveness of specific designs can vary across foundation models. Overall, GMA complements existing benchmarks by expanding application coverage and task complexity, providing a challenging testbed for evaluating mobile agents and studying how harness design supports reliable execution in complex mobile workflows. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-31 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computer Vision and Pattern Recognition (cs.CV). Pagina del lavoro: https://arxiv.org/abs/2608.27477 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T10:33:42.327Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).