Metraipolis › archive › dc_f749988e208d
MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models — arXiv:2608.03740
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.03740
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T12:16:38.093Z
- SHA-256 of the archived text
d5df0fe23a0cd50e0205be5f547f24f4dbb4c2872219a1b3bdcbda9b44bb7364- Stable URL of this page
https://metraipolis.com/documento/dc_f749988e208d
How to cite
MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models — arXiv:2608.03740 — arXiv (en), https://arxiv.org/abs/2608.03740 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f749988e208d (Metraipolis archive dc_f749988e208d, sha256 d5df0fe23a0cd50e0205be5f547f24f4dbb4c2872219a1b3bdcbda9b44bb7364)
BibTeX
@misc{dcf749988e208d,
title = {MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models — arXiv:2608.03740},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.03740},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f749988e208d; sha256 d5df0fe23a0cd50e0205be5f547f24f4dbb4c2872219a1b3bdcbda9b44bb7364},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03740. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Recent GUI visual grounding models generate screen coordinates as digit-token sequences that are parsed into numerical values and mapped to executable clicks. This generation-to-execution interface creates an attack surface that existing objectives over visual representations or coordinate-token sequences do not explicitly model. Although each coordinate digit is predicted as a token, its spatial effect after parsing depends on decimal position: changing a hundreds-place digit by one shifts the coordinate by 100 units, whereas the same change at the ones place shifts it by one. This mismatch motivates attack objectives that account for both numerical coordinate structure and click execution. Moreover, untargeted and targeted attacks require different objectives because they aim to move the click outside the correct region and into an attacker-specified region, respectively. We propose MissClick, an execution-aware white-box attack that aligns optimization with click-level success conditions. MissClick-U maximizes soft-coordinate displacement for untargeted disruption, while MissClick-T minimizes a place-weighted target-digit loss for targeted redirection. On OS-Atlas and UGround across desktop, web, and mobile platforms, MissClick-U achieves untargeted success rates of 75.07% and 72.93% (+16.62 and +30.72 pp), while MissClick-T achieves targeted success rates of 44.86% and 62.67% (+31.73 and +47.06 pp). Among the evaluated objectives, soft-coordinate displacement performs best for untargeted attacks, whereas place-weighted target-digit optimization performs best for targeted attacks, supporting goal-specific execution-aware objective design. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.03740 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T12:05:52.846Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).