Metraipolis › archive › dc_e30009916f8a

SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning — arXiv:2608.26432

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.26432
License (as archived)
CC0-1.0
Archived
2026-10-17T14:25:05.720Z
SHA-256 of the archived text
afd799f90b20980d907750dd28f8e4dc6488b69355fcecdfa7e104ba3c0f1888
Stable URL of this page
https://metraipolis.com/documento/dc_e30009916f8a

How to cite

SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning — arXiv:2608.26432 — arXiv (en), https://arxiv.org/abs/2608.26432 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e30009916f8a (Metraipolis archive dc_e30009916f8a, sha256 afd799f90b20980d907750dd28f8e4dc6488b69355fcecdfa7e104ba3c0f1888)
BibTeX
@misc{dce30009916f8a,
  title = {SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning — arXiv:2608.26432},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.26432},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e30009916f8a; sha256 afd799f90b20980d907750dd28f8e4dc6488b69355fcecdfa7e104ba3c0f1888},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26432. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Voice agents must call tools and hold multi-turn dialogue entirely through speech, yet the dominant paradigm trains them in text. Existing frameworks either cascade TTS and ASR around a proprietary voice API, where gradients cannot flow and per-call cost makes on-policy reinforcement learning prohibitive, or stay in text: they measure voice agents but cannot improve them. We present SpeechGym, an audio-native agentic environment in which two omni-modal models converse in native audio, with no external ASR or TTS and no API boundary, over the unmodified tasks, tools and success check of an established text agentic benchmark, so that the interaction modality is the only variable and the loop stays local and trainable end to end. Audio agentic capability does not follow from audio understanding. The failures speech introduces are perceptual rather than reasoning deficits: the agent picks the right tool and the right argument slot but fills it with a value misheard from the waveform, and that single error cascades into a failed call, a retry of the same call, and a wasted step budget. A second failure is behavioural: under an insistent caller the agent performs an unauthorised write and ends the episode believing it helped. Both are trainable, because the environment labels them for free: a call with a misheard argument fails against the database while a correct one succeeds. The obstacle is sparsity, not signal. Outcome-only GRPO is gradient-starved here, since almost every rollout group fails identically, while a per-turn process reward crediting each successful tool call restores variance to nearly every group. Trained this way, the agent transfers with no further tuning to an independently implemented voice benchmark, more than doubling task success and carrying an open-weights model from last place to second on that leaderboard, while using fewer turns and tokens than before training.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.AI).
Categorie arXiv: Sound (cs.SD) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.26432 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T13:16:53.640Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).