Metraipolis › archive › dc_4445bf9537fd

EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams — arXiv:2610.12248

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.12248
License (as archived): CC0-1.0
Archived: 2026-10-09T06:13:29.763Z
SHA-256 of the archived text: 4b2a0d9935e0a44ea25373e9c76c5172e93627e5fd0d02bfe6f2168694bca6bf
Stable URL of this page: https://metraipolis.com/documento/dc_4445bf9537fd

How to cite

EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams — arXiv:2610.12248 — arXiv (en), https://arxiv.org/abs/2610.12248 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4445bf9537fd (Metraipolis archive dc_4445bf9537fd, sha256 4b2a0d9935e0a44ea25373e9c76c5172e93627e5fd0d02bfe6f2168694bca6bf)

BibTeX

@misc{dc4445bf9537fd,
  title = {EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams — arXiv:2610.12248},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.12248},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4445bf9537fd; sha256 4b2a0d9935e0a44ea25373e9c76c5172e93627e5fd0d02bfe6f2168694bca6bf},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.12248. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Wearable augmented reality (AR) assistants are moving toward continuous real-world interaction, where they perceive the user's activity through first-person video and audio and provide timely spoken guidance without being explicitly asked. While proactive video assistants, spoken dialog systems, and egocentric task understanding have each advanced rapidly, existing systems do not address the joint problem of deciding when to speak and what to say from continuous first-person streams. We introduce EgoVoice, a framework for training and evaluating proactive egocentric spoken assistants. From HoloAssist video recordings of real human instructors, we construct clean audio streams through source separation and speech resynthesis, and convert each video session into a format where the model must decide at each moment whether to remain silent or provide spoken guidance. We fine-tune an omni-modal LLM with our data, and further improve its proactive intervention behavior with direct preference optimization. Experiments across closed and open-source models show that existing systems rarely produce well-timed, meaningful proactive interventions, while EgoVoice yields clear improvements in intervention timing, content relevance, and human preference over the zero-shot backbone.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Computer Vision and Pattern Recognition (cs.CV); Sound (cs.SD).
Pagina del lavoro: https://arxiv.org/abs/2610.12248 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T05:47:15.399Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).