Metraipolis › archive › dc_586034da052b

IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints — arXiv:2610.32536

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.32536
License (as archived)
CC0-1.0
Archived
2026-10-26T19:50:41.590Z
SHA-256 of the archived text
6697abc9592664f492682208b74da3fd0ccea9fe841afe9289352567e29e56ce
Stable URL of this page
https://metraipolis.com/documento/dc_586034da052b

How to cite

IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints — arXiv:2610.32536 — arXiv (en), https://arxiv.org/abs/2610.32536 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_586034da052b (Metraipolis archive dc_586034da052b, sha256 6697abc9592664f492682208b74da3fd0ccea9fe841afe9289352567e29e56ce)
BibTeX
@misc{dc586034da052b,
  title = {IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints — arXiv:2610.32536},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.32536},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_586034da052b; sha256 6697abc9592664f492682208b74da3fd0ccea9fe841afe9289352567e29e56ce},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.32536. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

GUI grounding benchmarks ask whether an agent can localize a target that is already visible in a static screenshot. Real interfaces often violate this assumption: a target may be off-screen, appear only after a state change or hover, remain occluded, or become actionable after a delay. This leaves the ability to recover target visibility unevaluated. To measure this reveal step, we introduce Interactive GUI Grounding (IGG), a benchmark in which an agent must first make the target visible through interaction and then click it. IGG contains 601 tasks across 16 controllable GUI environments and seven visibility-constraint sub-types, with a compact action space of scrolling, moving, dragging, waiting, and clicking. In our human study, participants achieve 94.6% accuracy, while GPT-5.5 reaches 79.4% and the strongest evaluated open-source VLM reaches only 31.6%. To identify the source of this gap, we conduct two controlled interventions. Telling Qwen3-VL which interaction to perform raises its accuracy only from 30.8% to 33.9%, whereas performing the interaction for the model raises it to 81.9%. The bottleneck is executing the reveal interaction, not recognizing the target once it is exposed. We further show that this capability is learnable, demonstrating that visibility recovery can be strengthened consistently across model families.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.32536 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-26T19:45:33.322Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).