Metraipolis › archive › dc_62bcaee2473c

Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis — arXiv:2609.03992

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.03992
License (as archived)
CC0-1.0
Archived
2026-10-17T01:40:54.578Z
SHA-256 of the archived text
3977c678c1aaea6dcdeb2d65f4edc9c1d0c02b5a445e70b9b2d6cf77e7bbbc45
Stable URL of this page
https://metraipolis.com/documento/dc_62bcaee2473c

How to cite

Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis — arXiv:2609.03992 — arXiv (en), https://arxiv.org/abs/2609.03992 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_62bcaee2473c (Metraipolis archive dc_62bcaee2473c, sha256 3977c678c1aaea6dcdeb2d65f4edc9c1d0c02b5a445e70b9b2d6cf77e7bbbc45)
BibTeX
@misc{dc62bcaee2473c,
  title = {Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis — arXiv:2609.03992},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.03992},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_62bcaee2473c; sha256 3977c678c1aaea6dcdeb2d65f4edc9c1d0c02b5a445e70b9b2d6cf77e7bbbc45},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03992. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We present Alignment-Free Text-Audiobox (Text-AB), a unified framework for high-quality voice dubbing and full-duplex dialogue synthesis. Building on a Diffusion Transformer trained with a flow-matching objective, Text-AB departs from the Audiobox system along three dimensions. First, it operates in a latent diffusion framework using DAC-VAE features that encode 48 kHz waveforms into a 25 Hz latent sequence, giving over 10x higher compression than previous EnCodec representations while improving resynthesis quality. Second, Text-AB is alignment-free: it consumes raw text via an off-the-shelf text encoder and learns text-speech alignment through cross-attention, removing the need for forced alignment and explicit duration prediction. Third, we scale model and data substantially, pretraining a 3B-parameter model on 480k hours of monolingual speech, followed by supervised fine-tuning on three downstream tasks: cross-lingual voice dubbing, full-duplex dialogue synthesis, and emotional full-duplex dialogue synthesis. At inference, Text-AB supports one-shot generation for up to ~1 min of speech and arbitrarily long-form generation via a multi-diffusion scheme, plus a multi-stage reranking strategy that enhances quality based on automated metrics. On a real-world dubbing benchmark, Text-AB delivers a step-change improvement over the latest internal dubbing system, with large gains in prosody similarity, voice similarity, naturalness, and shareability. For full-duplex dialogue synthesis, it approaches human recordings on short-form conversations and substantially outperforms the latest internal model on long-form human-likeness and expressivity, while natively modeling turn-taking, back-channeling, and emotional dynamics. For emotional dialogue synthesis, emotion conditioning significantly improves emotion alignment and emotional interaction quality over the unconditioned baseline.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Audio and Speech Processing (eess.AS).
Pagina del lavoro: https://arxiv.org/abs/2609.03992 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T01:11:43.570Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).