Metraipolis › archive › dc_f95c168776e2
A Multi-Modal Robotic Framework for Physical GUI Interaction Testing — arXiv:2611.06939
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.06939
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-06T14:51:23.749Z
- SHA-256 of the archived text
1d975dc262ffa07dc749ba700920ea32427d72d92ab3b6ac2b0e43a92e701235- Stable URL of this page
https://metraipolis.com/documento/dc_f95c168776e2
How to cite
A Multi-Modal Robotic Framework for Physical GUI Interaction Testing — arXiv:2611.06939 — arXiv (en), https://arxiv.org/abs/2611.06939 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f95c168776e2 (Metraipolis archive dc_f95c168776e2, sha256 1d975dc262ffa07dc749ba700920ea32427d72d92ab3b6ac2b0e43a92e701235)
BibTeX
@misc{dcf95c168776e2,
title = {A Multi-Modal Robotic Framework for Physical GUI Interaction Testing — arXiv:2611.06939},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.06939},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f95c168776e2; sha256 1d975dc262ffa07dc749ba700920ea32427d72d92ab3b6ac2b0e43a92e701235},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.06939. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Automated testing of mobile applications remains challenging when realistic user interactions must be reproduced and GUI behavior verified without relying on invasive mechanisms such as instrumentation or ADB-level control. Existing approaches, including DLD and R-DLD, detect data loss triggered by activity restarts but still depend on system-level access, which limits their applicability on real devices. This doctoral research proposes RFNIT, a multi-modal, fully non-invasive robotic framework for physical GUI interaction testing on smartphones. RFNIT supports end-to-end automation by perceiving the device state through external sensing and executing physical interactions, enabling human-like actions while preserving reproducibility and execution accuracy. The framework integrates a vision-based perception module, an LLM-driven reasoning agent, and a robotic actuator capable of performing touch, swipe, and typing gestures. Multimodal data, including external camera images and OCR-extracted text, are combined to identify GUI elements and guide subsequent testing decisions. Preliminary results indicate that RFNIT accurately recognizes buttons, text fields, and virtual keyboards from external camera input, achieving over 85\% visual detection accuracy. Ongoing and future work will focus on improving robustness, repeatability, and coverage of GUI states during long-running testing sessions. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2611.06939 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-06T14:21:57.526Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).