Metraipolis › archive › dc_d6796d63e07b
SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information — arXiv:2608.10692
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.10692
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T06:09:20.720Z
- SHA-256 of the archived text
4baf5b53d5606b6bd5f1663a4123b60148be33967adee3675794c78751b98d6d- Stable URL of this page
https://metraipolis.com/documento/dc_d6796d63e07b
How to cite
SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information — arXiv:2608.10692 — arXiv (en), https://arxiv.org/abs/2608.10692 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d6796d63e07b (Metraipolis archive dc_d6796d63e07b, sha256 4baf5b53d5606b6bd5f1663a4123b60148be33967adee3675794c78751b98d6d)
BibTeX
@misc{dcd6796d63e07b,
title = {SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information — arXiv:2608.10692},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.10692},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d6796d63e07b; sha256 4baf5b53d5606b6bd5f1663a4123b60148be33967adee3675794c78751b98d6d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.10692. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) are increasingly deployed as mobile assistants, where a key challenge is leveraging personal information scattered across multiple applications (apps) to complete user instructions. However, due to the lack of dedicated benchmarks, their capabilities remain poorly understood. To address this gap, we introduce SPIEval, a human-curated benchmark grounded in five cognitive capabilities (i.e., reasoning, disambiguation, integration, preference inference, and multi-intent decomposition). SPIEval comprises 250 tasks spanning 4,335 personal records distributed across 10 apps and supports multi-turn interaction through 21 tools. Analysis shows that the benchmark exhibits diverse scenarios, challenging tasks, scattered information, controllable environments, and verifiable outcomes. We evaluate nine representative LLMs and find substantial room for improvement. The best-performing model, GPT-5.5 (xhigh), achieves only 57.3% accuracy, while the weakest achieves just 16.4%. Further analysis reveals that 79% of failures stem from inaccurate information localization, as LLMs often commit to plausible but incorrect information instead of continuing retrieval for verification. We also find that fewer than 2% of retrieval actions employ advanced search methods and observe substantial variation in search efficiency across models. These findings expose fundamental limitations of current LLM-based mobile assistants and motivate future research in this direction. Data and code are available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-12 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.10692 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T05:57:28.395Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).