Metraipolis › archive › dc_f379579ccddf
From Visual Widgets to UI Code: Efficient Tool-Grounded Generation — arXiv:2608.12611
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.12611
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T04:27:47.640Z
- SHA-256 of the archived text
e46504e5bb84b4bdd2100885c6d19f23446f659814644e6ac8c40fbae38b0458- Stable URL of this page
https://metraipolis.com/documento/dc_f379579ccddf
How to cite
From Visual Widgets to UI Code: Efficient Tool-Grounded Generation — arXiv:2608.12611 — arXiv (en), https://arxiv.org/abs/2608.12611 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f379579ccddf (Metraipolis archive dc_f379579ccddf, sha256 e46504e5bb84b4bdd2100885c6d19f23446f659814644e6ac8c40fbae38b0458)
BibTeX
@misc{dcf379579ccddf,
title = {From Visual Widgets to UI Code: Efficient Tool-Grounded Generation — arXiv:2608.12611},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.12611},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f379579ccddf; sha256 e46504e5bb84b4bdd2100885c6d19f23446f659814644e6ac8c40fbae38b0458},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12611. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Existing screenshot-to-code systems face a trade-off between flexibility and controllability. Direct multimodal generation can hallucinate visible details, whereas structured pipelines reduce such errors through component-wise decomposition, predefined templates, and customized intermediate representations. These structures, however, introduce additional generative orchestration and restrict outputs to designs covered by the representation. We investigate whether selective tool grounding can improve the fidelity--efficiency trade-off of direct widget-to-code generation. We introduce \textbf{WidgetGen}, a lightweight tool-grounded framework that extracts observable text and color evidence, performs high-level layout and optional chart reasoning, and directly generates executable JavaScript XML (\emph{JSX}). This design reduces reliance on component-wise generation while avoiding a fixed UI schema. Across six multimodal models and \(1{,}000\) held-out widgets, WidgetGen outperforms direct prompting and the structured Widget2Code pipeline on most visual reconstruction metrics, with consistent gains in area, legibility, and style. Finally, reconstruction-derived image-code pairs improve six Qwen-family open-weight models across every reported metric through supervised fine-tuning. These results establish WidgetGen as a strong lightweight baseline and show that selective evidence grounding offers an effective alternative to extensive representation constraints.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-14 in cs.LG).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.12611 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T04:20:35.887Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).