Metraipolis › archive › dc_e47e047eefea

Decoupling What from Where: How Should a Small GUI Grounding Model Receive the Action Type? — arXiv:2610.07444

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.07444
License (as archived): CC0-1.0
Archived: 2026-10-07T11:41:08.975Z
SHA-256 of the archived text: 40a209debab7a8952c9c007cfbfc645bd2e1d7de5c0743dae9b2e2356bbf5571
Stable URL of this page: https://metraipolis.com/documento/dc_e47e047eefea

How to cite

Decoupling What from Where: How Should a Small GUI Grounding Model Receive the Action Type? — arXiv:2610.07444 — arXiv (en), https://arxiv.org/abs/2610.07444 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e47e047eefea (Metraipolis archive dc_e47e047eefea, sha256 40a209debab7a8952c9c007cfbfc645bd2e1d7de5c0743dae9b2e2356bbf5571)

BibTeX

@misc{dce47e047eefea,
  title = {Decoupling What from Where: How Should a Small GUI Grounding Model Receive the Action Type? — arXiv:2610.07444},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.07444},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e47e047eefea; sha256 40a209debab7a8952c9c007cfbfc645bd2e1d7de5c0743dae9b2e2356bbf5571},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07444. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A GUI agent decides which action to take and where to take it; we ask how a small grounding model should receive the action type. Fine-tuning Qwen2-VL-2B with LoRA on Android in the Wild, we compare a flat baseline with five ways of supplying the type under matched data, compute, and decoding: an auxiliary loss, a hard-routed action word, an additive learned embedding, a prepended learned token, and the type written into the prompt. With five seeds, an episode-clustered bootstrap, and seed-level paired tests, the ranking on a mixed stream is clear: the auxiliary loss, the additive embedding, and the prompt word each gain five to seven [email protected] points over the baseline, while hard routing and the prepended token are not distinguishable from it. Much of that gain is protection from a preprocessing choice of ours rather than a spatial prior. Our serializer clamps the off-screen touch point AITW records for type events to the origin; that class degrades the baseline's click grounding, and removing it lifts the baseline by nearly seven points, after which no mechanism's hit rate beats it and the intervals exclude a two-point effect, though the auxiliary loss still shortens the average miss; on a stream of taps and swipes none helps. Whether this generalizes beyond one serialization is open. For deployment, the pipeline's margin over the baseline with predicted rather than gold types is not established (+0.016, 95% interval [-0.017, +0.052]), and a wrong type collapses every model conditioned at inference. The prepended token does not help at the shared learning rate, where its rows barely move from initialization; trained ten times faster it reaches the level of the other three, with a margin three seeds do not establish. We also document a silent failure: injecting conditioning through inputs_embeds makes Qwen2-VL fall back to 1-D positions for image tokens, costing nine points.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2610.07444 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:19:24.759Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).