Metraipolis › archive › dc_e3ca34ce7158

SURE-EVAL: A Systematic and Unified Agentic Framework for Reproducible Evaluation — arXiv:2609.39030

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.39030
License (as archived)
CC0-1.0
Archived
2026-10-07T16:18:43.278Z
SHA-256 of the archived text
ab8e4dc43b29f782bf660cc74b4b16b510cafc2d636df9b192043d3aadc349df
Stable URL of this page
https://metraipolis.com/documento/dc_e3ca34ce7158

How to cite

SURE-EVAL: A Systematic and Unified Agentic Framework for Reproducible Evaluation — arXiv:2609.39030 — arXiv (en), https://arxiv.org/abs/2609.39030 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e3ca34ce7158 (Metraipolis archive dc_e3ca34ce7158, sha256 ab8e4dc43b29f782bf660cc74b4b16b510cafc2d636df9b192043d3aadc349df)
BibTeX
@misc{dce3ca34ce7158,
  title = {SURE-EVAL: A Systematic and Unified Agentic Framework for Reproducible Evaluation — arXiv:2609.39030},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.39030},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e3ca34ce7158; sha256 ab8e4dc43b29f782bf660cc74b4b16b510cafc2d636df9b192043d3aadc349df},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39030. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Audio and speech models are released rapidly, but reported scores often conflate model capability with deployment and evaluation choices. The same checkpoint can produce different predictions under different runtimes, hardware, decoding settings, or fallback policies. Even fixed predictions can receive different scores under different normalization and metric implementations. Existing speech benchmarks standardize selected datasets or scoring procedures, but rarely connect heterogeneous model onboarding, controlled inference, and versioned scoring in one executable workflow. We introduce SURE-EVAL, a Systematic and Unified Agentic framework for Reproducible Evaluation of audio and speech systems. A Tool Agent Workflow converts model releases into isolated, verified callable tools. A Main Agent Workflow commits task-specific inference and scoring protocols, then delegates all score-bearing operations to versioned deterministic programs. Each result retains its runtime, protocol, pipeline nodes, predictions, and audit artifacts. Across 18 public releases covering automatic speech recognition, text-to-speech, voice conversion, speaker diarization, speaker-attributed recognition, and multi-task audio understanding, a Codex-only baseline completes 12 models in one shot, while the same agent with the SURE-EVAL Tool Agent Workflow completes all 18. We also conduct unified evaluations over seven ASR test conditions and two TTS subsets. A protocol analysis of three TTS systems finds absolute differences of 0.02-0.52 points between paper-reported and unified results, with the direction varying by model and language. These results show that reproducible evaluation requires controlling both model execution and output scoring.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in eess.AS).
Categorie arXiv: Audio and Speech Processing (eess.AS).
Pagina del lavoro: https://arxiv.org/abs/2609.39030 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:51:53.224Z dalla pagina https://arxiv.org/catchup/eess.AS/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).