Metraipolis › archive › dc_be376e0139cd

SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation — arXiv:2608.10775

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.10775
License (as archived): CC0-1.0
Archived: 2026-10-19T06:08:36.286Z
SHA-256 of the archived text: 095b71f610ec869809b37f6154757feaf7c1a02b3e4d71be8f3abdd7f7d3f23b
Stable URL of this page: https://metraipolis.com/documento/dc_be376e0139cd

How to cite

SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation — arXiv:2608.10775 — arXiv (en), https://arxiv.org/abs/2608.10775 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_be376e0139cd (Metraipolis archive dc_be376e0139cd, sha256 095b71f610ec869809b37f6154757feaf7c1a02b3e4d71be8f3abdd7f7d3f23b)

BibTeX

@misc{dcbe376e0139cd,
  title = {SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation — arXiv:2608.10775},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.10775},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_be376e0139cd; sha256 095b71f610ec869809b37f6154757feaf7c1a02b3e4d71be8f3abdd7f7d3f23b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.10775. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Computer-using agents can perceive rich software interfaces, yet their decisions often lack visual procedural memory: they may recognize individual controls without identifying which familiar workflow is active, which control matters next, or what evidence would confirm progress. Raw interaction traces preserve such information but are long and noisy to condition on, whereas text-only skills often omit the visual state that makes a procedure applicable. We introduce Visual Skill Cards (VSCs), a state-conditioned memory representation that binds reusable procedures with applicability cues, visual evidence, and verification signals. SkillLens constructs VSCs from heterogeneous interaction experience through Trace-to-Visual-Skill-Card and, at inference time, retrieves relevant cards and selectively expands only the evidence needed by a fixed visual-language model executor for grounded GUI action prediction. The same representation also supports CardDistill, which uses VSC evidence as privileged teacher context to train a student that acts without runtime card retrieval. Across Multimodal-Mind2Web and WebLINX-BrowserGym, SkillLens improves the frozen GPT-5.4-mini executor by +11.6 points in Step SR and +2.9 points in Overall, respectively; CardDistill further improves the corresponding student-only Qwen3-VL-2B metrics by +12.0 and +3.2 points.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-12 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.10775 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:57:28.296Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).