Metraipolis › archive › dc_06cccdaa0b85

TBDub: Production-Oriented Visual Dubbing — arXiv:2609.06144

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.06144
License (as archived)
CC0-1.0
Archived
2026-10-16T15:06:50.730Z
SHA-256 of the archived text
9dc691fe321ef2585b5021ea9d4ed5922972a8c1c70f156e29bb21cfe8a1a2c4
Stable URL of this page
https://metraipolis.com/documento/dc_06cccdaa0b85

How to cite

TBDub: Production-Oriented Visual Dubbing — arXiv:2609.06144 — arXiv (en), https://arxiv.org/abs/2609.06144 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_06cccdaa0b85 (Metraipolis archive dc_06cccdaa0b85, sha256 9dc691fe321ef2585b5021ea9d4ed5922972a8c1c70f156e29bb21cfe8a1a2c4)
BibTeX
@misc{dc06cccdaa0b85,
  title = {TBDub: Production-Oriented Visual Dubbing — arXiv:2609.06144},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.06144},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_06cccdaa0b85; sha256 9dc691fe321ef2585b5021ea9d4ed5922972a8c1c70f156e29bb21cfe8a1a2c4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.06144. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Visual dubbing must synchronize mouth motion with replacement speech while preserving identity, appearance, and temporal consistency. Although X-Dub provides a strong mask-free video-editing baseline, its application to livestream and generated-video content reveals limitations in production-domain robustness, temporal and motion stability, identity and oral-detail preservation, and inference efficiency. We present \textbf{TBDub}, a production-oriented extension of X-Dub that combines task-adaptive post-training with task-aware few-step distillation. Post-training adapts the video DiT using production-domain data, production-specific conditioning and filtering, and enhanced audio features to obtain a 30-step Teacher. Distillation adapts DMD/DMD2 to conditional video editing and compresses the Teacher into a two-step Student. On 38 TalkVid clips, the Teacher improves all eight reported reconstruction, perceptual, identity, and synchronization metrics over X-Dub. In the MOS evaluation, it improves lip-sync consistency, identity consistency, and visual quality over X-Dub by 0.14, 0.95, and 0.90 points, while the Student achieves the highest lip-sync and visual-quality scores and remains close to the Teacher in identity consistency. In paired end-to-end generation timing from the first VAE encode through the final VAE decode on a single NVIDIA H20 GPU at $512\times512$, the Student reaches 7.13 effective FPS and reduces total latency by $13.93\times$; the DiT stage alone is accelerated by $42.49\times$. The Student largely retains the Teacher's generation quality and audiovisual synchronization. The code is available on GitHub at \ this https URL , and the 30-step Teacher and two-step Student weights are available on Hugging Face at this https URL .

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CV).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2609.06144 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T14:49:57.049Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).