Metraipolis › archive › dc_bc95acfe669d

Reference-Based Manipulation: A Framework and Pipeline for Multimodal Spatial Reasoning — arXiv:2608.04798

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.04798
License (as archived): CC0-1.0
Archived: 2026-10-31T11:28:28.399Z
SHA-256 of the archived text: 691c78745f4328ea179dd15abbbac1f12879c0cb77833d8396c67ca16095d447
Stable URL of this page: https://metraipolis.com/documento/dc_bc95acfe669d

How to cite

Reference-Based Manipulation: A Framework and Pipeline for Multimodal Spatial Reasoning — arXiv:2608.04798 — arXiv (en), https://arxiv.org/abs/2608.04798 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bc95acfe669d (Metraipolis archive dc_bc95acfe669d, sha256 691c78745f4328ea179dd15abbbac1f12879c0cb77833d8396c67ca16095d447)

BibTeX

@misc{dcbc95acfe669d,
  title = {Reference-Based Manipulation: A Framework and Pipeline for Multimodal Spatial Reasoning — arXiv:2608.04798},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.04798},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bc95acfe669d; sha256 691c78745f4328ea179dd15abbbac1f12879c0cb77833d8396c67ca16095d447},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.04798. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

When manipulating objects in immersive platforms through speech and gesture, users naturally construct spatial references, referring to scene entities, their bodies, or the environment. Leveraging spatial cognition theories, this work systematically examines how users construct and communicate spatial intent. Using a custom toolkit, we conducted a Wizard-of-Oz study to observe unconstrained multimodal (speech + gesture) input patterns in Virtual Reality for scene construction. Based on these findings, we formalize a framework that decomposes spatial references into three core components: Source, Anchor, and Frame, while characterizing their compositional strategies and explicitness. We demonstrate the utility of this Reference-based Manipulation framework by implementing an LLM-based pipeline featuring a set of example interaction techniques with a preliminary technical evaluation. Finally, we discuss key lessons learned for supporting reference-based spatial interaction.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.HC).
Categorie arXiv: Human-Computer Interaction (cs.HC).
Pagina del lavoro: https://arxiv.org/abs/2608.04798 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T11:17:34.224Z dalla pagina https://arxiv.org/catchup/cs.HC/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).