Metraipolis › archive › dc_dce00fe4313f

Listen, Critique, and Refine: RL-Based Self-Refinement for Instruction-Following Speech Synthesis — arXiv:2609.24163

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.24163
License (as archived)
CC0-1.0
Archived
2026-10-10T01:00:37.353Z
SHA-256 of the archived text
7cf82041f323a08058258448aae1c4de446cf6f4a4a24b0c6e113de59f60e4f4
Stable URL of this page
https://metraipolis.com/documento/dc_dce00fe4313f

How to cite

Listen, Critique, and Refine: RL-Based Self-Refinement for Instruction-Following Speech Synthesis — arXiv:2609.24163 — arXiv (en), https://arxiv.org/abs/2609.24163 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dce00fe4313f (Metraipolis archive dc_dce00fe4313f, sha256 7cf82041f323a08058258448aae1c4de446cf6f4a4a24b0c6e113de59f60e4f4)
BibTeX
@misc{dcdce00fe4313f,
  title = {Listen, Critique, and Refine: RL-Based Self-Refinement for Instruction-Following Speech Synthesis — arXiv:2609.24163},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.24163},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dce00fe4313f; sha256 7cf82041f323a08058258448aae1c4de446cf6f4a4a24b0c6e113de59f60e4f4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.24163. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large Audio Language Models (LALMs) can follow diverse instructions to synthesize speech in specified styles. However, complex instructions that require simultaneous control over pitch dynamics, speaking rate, and emotional tone often exceed what a single-pass generation can faithfully realize. While recent reasoning models have shown that intermediate "thinking" tokens improve output quality, this paradigm has been confined to the text modality. In this work, we extend reasoning to the audio token space by training a LALM with reinforcement learning to reason over its own speech output. The model first generates a draft speech as a form of audio-token reasoning, critiques its own generation by reflecting on the acoustic realization in text, and then produces a refined version conditioned on both the first-pass speech and the critique, all within a single model. After RL training, the refined two-hop outputs achieve a relative improvement of 7.15\% on the InstructTTSEval benchmark, demonstrating the model's reflective ability.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in eess.AS).
Categorie arXiv: Audio and Speech Processing (eess.AS) ; Sound (cs.SD).
Pagina del lavoro: https://arxiv.org/abs/2609.24163 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T00:12:36.403Z dalla pagina https://arxiv.org/catchup/eess.AS/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).