Metraipolis › archive › dc_dc3a7e656762
Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding — arXiv:2608.26925
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.26925
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T15:20:03.718Z
- SHA-256 of the archived text
8933deef64d75ce7b532edf6efbf37b41f3549dc666048a5cb6b65e8cb33476c- Stable URL of this page
https://metraipolis.com/documento/dc_dc3a7e656762
How to cite
Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding — arXiv:2608.26925 — arXiv (en), https://arxiv.org/abs/2608.26925 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dc3a7e656762 (Metraipolis archive dc_dc3a7e656762, sha256 8933deef64d75ce7b532edf6efbf37b41f3549dc666048a5cb6b65e8cb33476c)
BibTeX
@misc{dcdc3a7e656762,
title = {Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding — arXiv:2608.26925},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.26925},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dc3a7e656762; sha256 8933deef64d75ce7b532edf6efbf37b41f3549dc666048a5cb6b65e8cb33476c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26925. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
In many low-resource settings, even just eliciting speech for data collection is difficult. One promising approach has been to ask speakers to describe images. But how do we build models from such visually grounded speech data? Given a dataset of images with Hindi spoken captions, we consider how we can map a written English keyword to spoken realisations of that word in Hindi. Previous work trained end-to-end multimodal neural models. Instead, we explore a simpler alignment-based approach built on self-supervised speech representations. Written English tags are automatically obtained from images using off-the-shelf image captioning systems. Hindi utterances associated with the same keyword are then aligned (using self-supervised features), and alignment evidence is aggregated to identify recurring speech segments corresponding to the target word. Experiments evaluating keyword spotting and localization show that our alignment-based approach outperforms a previous attention-based neural model. We also show the benefit of incorporating negative examples during alignment. Our work demonstrates that cross-lingual word-to-speech mappings can be learned directly from visual grounding without transcriptions or explicit model training. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Audio and Speech Processing (eess.AS). Pagina del lavoro: https://arxiv.org/abs/2608.26925 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T15:15:18.077Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).