Metraipolis › archive › dc_0ceb08cbb87b

MTVA-Bench: Evaluating the Language Model Inside Cascaded Voice Agents — arXiv:2609.20152

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.20152
License (as archived)
CC0-1.0
Archived
2026-10-10T03:21:43.278Z
SHA-256 of the archived text
aff6aab7256623a47013b8ad9d2f39418db0491e642bf9cc64641b05f9ede464
Stable URL of this page
https://metraipolis.com/documento/dc_0ceb08cbb87b

How to cite

MTVA-Bench: Evaluating the Language Model Inside Cascaded Voice Agents — arXiv:2609.20152 — arXiv (en), https://arxiv.org/abs/2609.20152 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0ceb08cbb87b (Metraipolis archive dc_0ceb08cbb87b, sha256 aff6aab7256623a47013b8ad9d2f39418db0491e642bf9cc64641b05f9ede464)
BibTeX
@misc{dc0ceb08cbb87b,
  title = {MTVA-Bench: Evaluating the Language Model Inside Cascaded Voice Agents — arXiv:2609.20152},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.20152},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0ceb08cbb87b; sha256 aff6aab7256623a47013b8ad9d2f39418db0491e642bf9cc64641b05f9ede464},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.20152. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Generally, most voice agents are cascaded systems, i.e., an ASR model transcribes the caller's audio, a language model reads the transcript and decides what to say and which backend tools to call, and a TTS model speaks the reply. Nearly all of the decision making happens in the language model, but existing evaluations measure it either too broadly or too narrowly. End-to-end voice benchmarks score the full pipeline, so recognition errors and model errors mix into a single number. LLM benchmarks isolate the model but they do not evaluate what makes real phone calls hard, such as transcription issues, caller's voice being split across messages and the requirement that replies follow the language and script specified. We introduce the Multi-Turn Voice Agent Benchmark (MTVA-Bench), which evaluates the language model on the same conditions it faces inside a cascaded system. The caller is played by an LLM following a set of rubrics and tool calls are answered by a mock backend which responds to the arguments the model actually sent. The benchmark contains 49 agents working across 490 reviewed scenarios and supports 7 languages. Scoring is a combination of deterministic checks on tool calls with two LLM judges, one that scores scenario specific rules and one that grades conversation quality without access to the task. Both judges must cite specific messages from the transcript. Task and conversation scores are weighted equally, since a call can complete its task and still go badly for the caller. In a seven-model study, six of the models select the correct tool within 6.4 points of one another, but their overall scores span 24.4 points. Most of the gap comes from argument values, action ordering, rule compliance, and what the model says around its tool calls.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-18 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.20152 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T03:04:13.713Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).