Metraipolis › archive › dc_98c98985d4e1

Monotonicity-Guided Semantic Alignment for Zero-shot Multispeaker Image-to-Speech Synthesis — arXiv:2609.38440

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.38440
License (as archived)
CC0-1.0
Archived
2026-10-07T16:18:40.136Z
SHA-256 of the archived text
4837dc35e7d501e2ec3a687511facaf001a4eb77e0fb7a66f73179266a65b506
Stable URL of this page
https://metraipolis.com/documento/dc_98c98985d4e1

How to cite

Monotonicity-Guided Semantic Alignment for Zero-shot Multispeaker Image-to-Speech Synthesis — arXiv:2609.38440 — arXiv (en), https://arxiv.org/abs/2609.38440 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_98c98985d4e1 (Metraipolis archive dc_98c98985d4e1, sha256 4837dc35e7d501e2ec3a687511facaf001a4eb77e0fb7a66f73179266a65b506)
BibTeX
@misc{dc98c98985d4e1,
  title = {Monotonicity-Guided Semantic Alignment for Zero-shot Multispeaker Image-to-Speech Synthesis — arXiv:2609.38440},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.38440},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_98c98985d4e1; sha256 4837dc35e7d501e2ec3a687511facaf001a4eb77e0fb7a66f73179266a65b506},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38440. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Direct image-to-speech (Img2Sp) poses an alignment challenge in mapping visual content to ordered speech sequences, as images permit multiple spoken descriptions and lack monotonic correspondence with speech sequences. We propose Monotonicity-Guided Semantic Alignment (MGSA), to the best of our knowledge, the first framework for zero-shot multispeaker Img2Sp synthesis. We use semantic speech units to provide shared content targets across speakers with reference speech for speaker conditioning. A query aligner maps semantic memory learned from visual content to speech unit positions via a soft monotonic prior, which yields position-specific conditioning states. A blockwise masked diffusion generator is employed for the speech unit generation conditioning on these states. Experiments on Flickr8k-Audio show competitive captioning performance against single-speaker baselines, while evaluation with LibriTTS-R references supports zero-shot synthesis for unseen speakers. Ablations validate the effectiveness of aligner and block diffusion. Audio samples are available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in eess.AS).
Categorie arXiv: Audio and Speech Processing (eess.AS).
Pagina del lavoro: https://arxiv.org/abs/2609.38440 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:51:53.218Z dalla pagina https://arxiv.org/catchup/eess.AS/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).