Metraipolis › archive › dc_0cfbf05c98ef

Learning Timeline-Preserving Speech-to-Speech Translation — arXiv:2611.00690

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.00690
License (as archived)
CC0-1.0
Archived
2026-11-03T18:24:32.749Z
SHA-256 of the archived text
77c448f1a86b5a27c06c9e69028c810d9a6895278e0050f1e8a02edea6485a3f
Stable URL of this page
https://metraipolis.com/documento/dc_0cfbf05c98ef

How to cite

Learning Timeline-Preserving Speech-to-Speech Translation — arXiv:2611.00690 — arXiv (en), https://arxiv.org/abs/2611.00690 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0cfbf05c98ef (Metraipolis archive dc_0cfbf05c98ef, sha256 77c448f1a86b5a27c06c9e69028c810d9a6895278e0050f1e8a02edea6485a3f)
BibTeX
@misc{dc0cfbf05c98ef,
  title = {Learning Timeline-Preserving Speech-to-Speech Translation — arXiv:2611.00690},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.00690},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0cfbf05c98ef; sha256 77c448f1a86b5a27c06c9e69028c810d9a6895278e0050f1e8a02edea6485a3f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00690. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Speech-to-speech translation (S2ST) increasingly preserves what is said and how it sounds, but not necessarily when speech and silence occur. For dubbing and other temporally grounded applications, matching total duration is not enough: speech and pauses must also occur at corresponding moments. We formulate timeline-preserving S2ST, which aims to preserve the source endpoint and salient internal pauses on a shared absolute timeline. We present TransDubber, an end-to-end autoregressive system that learns this temporal structure directly from aligned target waveforms, without inference-time frame-level timing control or post-hoc temporal correction. To provide this supervision, we construct 642K exactly duration-matched English-Chinese speech pairs by translating and synthesizing target speech on source-defined temporal scaffolds. Training combines a progressive supervised curriculum with verifier-guided reinforcement learning to balance translation quality and temporal alignment. On a 4,353-utterance test set spanning read and spontaneous speech, TransDubber increases the proportion of translations within 2% of the source duration from 25.6% to 61.5% and pause-alignment F1 from 0.118 to 0.550 compared with UniSS, while maintaining competitive translation quality. Our analysis shows that aligned supervision establishes timeline following, while reinforcement learning refines temporal precision and partially offsets the translation-quality cost of strict synchronization. Together, these results demonstrate the effectiveness of our data and training framework for timeline-preserving S2ST. Audio samples, code, datasets, and pretrained models are available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Audio and Speech Processing (eess.AS) ; Computation and Language (cs.CL); Sound (cs.SD).
Pagina del lavoro: https://arxiv.org/abs/2611.00690 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:10:08.019Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).