Metraipolis › archive › dc_598fe6ff811d

Louder, Longer, Livelier: Acoustic Shortcuts and Underspecified Rationales in Speech LLM Judges — arXiv:2609.36979

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.36979
License (as archived)
CC0-1.0
Archived
2026-10-07T20:52:58.560Z
SHA-256 of the archived text
82373721729e5f9f589d60d7fffc63f6cee31999c256f4766034a6a75f14d3e6
Stable URL of this page
https://metraipolis.com/documento/dc_598fe6ff811d

How to cite

Louder, Longer, Livelier: Acoustic Shortcuts and Underspecified Rationales in Speech LLM Judges — arXiv:2609.36979 — arXiv (en), https://arxiv.org/abs/2609.36979 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_598fe6ff811d (Metraipolis archive dc_598fe6ff811d, sha256 82373721729e5f9f589d60d7fffc63f6cee31999c256f4766034a6a75f14d3e6)
BibTeX
@misc{dc598fe6ff811d,
  title = {Louder, Longer, Livelier: Acoustic Shortcuts and Underspecified Rationales in Speech LLM Judges — arXiv:2609.36979},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.36979},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_598fe6ff811d; sha256 82373721729e5f9f589d60d7fffc63f6cee31999c256f4766034a6a75f14d3e6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36979. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM-as-a-judge is widely used for evaluating text, but extending this paradigm to speech requires models to interpret acoustic as well as linguistic evidence. This introduces a modality-specific risk: a speech judge may treat a perceptually salient cue as evidence of quality even when that cue is irrelevant to the target criterion or receives more weight than human listeners give it. We call this behavior an acoustic shortcut. To study it, we audit six speech LLM judges using controlled manipulations of intensity, content richness, and emotional delivery. We evaluate both pointwise scoring and pairwise comparison, using human preference calibration to interpret the results. The judges consistently reward louder audio, prefer content-rich speech more strongly than human listeners do, and map emotional delivery into quality preferences. These effects are most visible in pairwise comparison, while pointwise scores often obscure them. More concerningly, the accompanying rationales rarely identify the acoustic cue that changes a judgment and instead repeatedly rely on a limited vocabulary, leaving them acoustically underspecified. Together, these findings show that reliable speech judges must both resist acoustic shortcuts and ground their rationales in the acoustic evidence behind their decisions. To support reproducibility and future audits, we also release SpeechJudgeAudit, the controlled stimuli and evaluation tools used in this study.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in eess.AS).
Categorie arXiv: Audio and Speech Processing (eess.AS) ; Sound (cs.SD).
Pagina del lavoro: https://arxiv.org/abs/2609.36979 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:18:02.982Z dalla pagina https://arxiv.org/catchup/eess.AS/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).