Metraipolis › archive › dc_e5bf42113e2d
Kraken: LLM-based Speech-to-Speech Translation via Low-bitrate VQ and Dual-path Source Conditioning — arXiv:2609.13045
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.13045
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T03:08:28.626Z
- SHA-256 of the archived text
d86b7ecc53f1c0500c7a69dcb3f629db261e10bad9eaf443c23234dbdce0d1ea- Stable URL of this page
https://metraipolis.com/documento/dc_e5bf42113e2d
How to cite
Kraken: LLM-based Speech-to-Speech Translation via Low-bitrate VQ and Dual-path Source Conditioning — arXiv:2609.13045 — arXiv (en), https://arxiv.org/abs/2609.13045 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e5bf42113e2d (Metraipolis archive dc_e5bf42113e2d, sha256 d86b7ecc53f1c0500c7a69dcb3f629db261e10bad9eaf443c23234dbdce0d1ea)
BibTeX
@misc{dce5bf42113e2d,
title = {Kraken: LLM-based Speech-to-Speech Translation via Low-bitrate VQ and Dual-path Source Conditioning — arXiv:2609.13045},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.13045},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e5bf42113e2d; sha256 d86b7ecc53f1c0500c7a69dcb3f629db261e10bad9eaf443c23234dbdce0d1ea},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.13045. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Speech-to-speech translation (S2ST) has advanced significantly with speech LLMs, offering the potential for joint optimization and preserving non-linguistic information. However, these models struggle with predicting high-bitrate speech tokens in LLMs, and face the challenge of relying on S2ST training data with ideally aligned speaker identity and prosody. We propose using low-bitrate tokens based on single-layer vector quantization, trained to reconstruct self-supervised learning (SSL) features. We also employ a separate token-to-waveform decoder named Autowave-X, which is also conditioned on the source speech to improve non-linguistic transfer, thereby relaxing the training data constraints. With the integration of these techniques, we propose an S2ST model named Kraken, which augments a pre-trained LLM with speech feature inputs and the low-bitrate token outputs, followed by Autowave-X vocoder. We built the model upon Qwen3-8B and trained it using 150k hours of multilingual and multitask speech data. We demonstrated that our model exhibited better translation quality than SeamlessM4T-Large v2 and Qwen2.5-Omni, along with improved speaker and prosody transfer capabilities. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-14 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Sound (cs.SD). Pagina del lavoro: https://arxiv.org/abs/2609.13045 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T03:02:47.554Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).