Metraipolis › archive › dc_dacfa100a12c

Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer — arXiv:2608.15690

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.15690
License (as archived)
CC0-1.0
Archived
2026-10-18T18:04:15.826Z
SHA-256 of the archived text
12dc7a4dce3119d43768d2c01abff653199583d832d2fd1730ea8d341614e4f5
Stable URL of this page
https://metraipolis.com/documento/dc_dacfa100a12c

How to cite

Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer — arXiv:2608.15690 — arXiv (en), https://arxiv.org/abs/2608.15690 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dacfa100a12c (Metraipolis archive dc_dacfa100a12c, sha256 12dc7a4dce3119d43768d2c01abff653199583d832d2fd1730ea8d341614e4f5)
BibTeX
@misc{dcdacfa100a12c,
  title = {Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer — arXiv:2608.15690},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.15690},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dacfa100a12c; sha256 12dc7a4dce3119d43768d2c01abff653199583d832d2fd1730ea8d341614e4f5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.15690. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Text-to-audio-video (T2AV) generation models produce a video and its soundtrack from a textual description, but offer no control over whose voice speaks in the output. We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-initialized linear layer on top of its audio backbone, fine-tuning for a comparatively short training schedule, and conditioning on a short reference recording at inference time. The reference is injected through two complementary signals: its diffusion latents are prepended to the audio stream, and a global speaker embedding modulates token of the target audio. On a benchmark of 674 speaker-text pairs spanning 30 speakers we compare against five strong voice-cloning text-to-speech baselines: our enhanced 5B model attains the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming every baseline. A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a ~30x speed-up over the full audio-video diffusion loop while preserving the voice-cloning behaviour.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI).
Categorie arXiv: Sound (cs.SD) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multimedia (cs.MM).
Pagina del lavoro: https://arxiv.org/abs/2608.15690 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T16:29:14.043Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).