Metraipolis › archive › dc_336ea37a9476

Navigation Alone Is Not Enough: Evaluating Explanatory Assistive UI Agents — arXiv:2608.09944

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.09944
License (as archived)
CC0-1.0
Archived
2026-10-19T06:08:46.413Z
SHA-256 of the archived text
9b18159f1c4c72c66fc673c105a165c0aa7522e03258ee0fc276c8aee2262c79
Stable URL of this page
https://metraipolis.com/documento/dc_336ea37a9476

How to cite

Navigation Alone Is Not Enough: Evaluating Explanatory Assistive UI Agents — arXiv:2608.09944 — arXiv (en), https://arxiv.org/abs/2608.09944 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_336ea37a9476 (Metraipolis archive dc_336ea37a9476, sha256 9b18159f1c4c72c66fc673c105a165c0aa7522e03258ee0fc276c8aee2262c79)
BibTeX
@misc{dc336ea37a9476,
  title = {Navigation Alone Is Not Enough: Evaluating Explanatory Assistive UI Agents — arXiv:2608.09944},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.09944},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_336ea37a9476; sha256 9b18159f1c4c72c66fc673c105a165c0aa7522e03258ee0fc276c8aee2262c79},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.09944. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Modern web interfaces are increasingly difficult to use with screen readers, particularly when pages update dynamically or hide important structure behind visual layout. Recent UI agents can act on such interfaces; however, for assistive agents to be truly useful, they must behave as collaborators that keep users informed and in control, rather than as tools that simply take actions on users' behalf. Most existing benchmarks judge systems primarily by task completion, without assessing how well they explain their actions or support user oversight. We introduce NeXUI, a benchmark for assistive agents that must navigate interfaces while explaining each step in clear language for nonvisual use. NeXUI pairs realistic user goals with instrumented interface states, enabling agents to reason from both visual context and structural information. Its evaluation measures safety, efficiency, and task success, while also checking whether explanations are grounded in the interface state. In our experiments, we find that NeXUI remains challenging even for state-of-the-art foundation models, with % Gemini-3.5-Flash achieving only a 44\% success rate and poor explanation scores, making it a useful foundation for future research and development. By focusing on navigation, explanation, and user control, NeXUI provides a clearer way to study agents that can support blind and visually impaired users in modern computing environments.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-12 in cs.AI).
Categorie arXiv: Human-Computer Interaction (cs.HC) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.09944 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:57:28.318Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).