Metraipolis › archive › dc_cd38e8f0b7e8
Learning Natural Conversational Behavior in Tandem Speech-to-Speech Models with Randomized Guidance — arXiv:2609.30773
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.30773
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T02:03:58.699Z
- SHA-256 of the archived text
57132dfc26877a6238ef3fb737ddc32bde3f1fe209097fccb71b24a11c1c656e- Stable URL of this page
https://metraipolis.com/documento/dc_cd38e8f0b7e8
How to cite
Learning Natural Conversational Behavior in Tandem Speech-to-Speech Models with Randomized Guidance — arXiv:2609.30773 — arXiv (en), https://arxiv.org/abs/2609.30773 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cd38e8f0b7e8 (Metraipolis archive dc_cd38e8f0b7e8, sha256 57132dfc26877a6238ef3fb737ddc32bde3f1fe209097fccb71b24a11c1c656e)
BibTeX
@misc{dccd38e8f0b7e8,
title = {Learning Natural Conversational Behavior in Tandem Speech-to-Speech Models with Randomized Guidance — arXiv:2609.30773},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.30773},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cd38e8f0b7e8; sha256 57132dfc26877a6238ef3fb737ddc32bde3f1fe209097fccb71b24a11c1c656e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.30773. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Tandem speech-to-speech architectures couple a responsive speech frontend with an asynchronous text backend. In KAME, a large language model (LLM) serves as the backend, supplying candidate responses as guidance to the speech frontend while the user is still speaking. Ordinary conversation recordings capture the eventual response but not the guidance the backend would supply during the user's utterance. Generating the missing guidance with a simulator LLM adds substantial data-preparation overhead when training on real conversations. We propose randomized intermediate guidance, which derives guidance directly from the conversation corpus rather than simulating backend LLM behavior. During training, target responses provide informative guidance, while randomly sampled responses provide potentially irrelevant updates during the utterance. This combination aims to teach the frontend to use backend information selectively. On synthetic dialogues, KAME trained with this recipe achieves response quality comparable to that of the LLM-generated and similarity-based baselines. Training on 3.8k hours of real conversations improves smooth turn-taking and audio-judge naturalness over synthetic-data KAME while retaining a response-quality advantage over Moshi. These results show that randomized guidance offers a practical route to combining the response-quality benefits of tandem models with natural conversational behavior learned from real speech. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-28 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Audio and Speech Processing (eess.AS). Pagina del lavoro: https://arxiv.org/abs/2609.30773 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T01:38:49.438Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).