Metraipolis › archive › dc_81adca3429ce

SITAR: A Situational Distortion Metric for Automatic Speech Recognition Evaluation — arXiv:2610.21440

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.21440
License (as archived)
CC0-1.0
Archived
2026-10-16T09:57:41.757Z
SHA-256 of the archived text
a5ef7d543adad872cf5f8e6405297500e11fd491b3e9f19c506c01fcea8ca396
Stable URL of this page
https://metraipolis.com/documento/dc_81adca3429ce

How to cite

SITAR: A Situational Distortion Metric for Automatic Speech Recognition Evaluation — arXiv:2610.21440 — arXiv (en), https://arxiv.org/abs/2610.21440 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_81adca3429ce (Metraipolis archive dc_81adca3429ce, sha256 a5ef7d543adad872cf5f8e6405297500e11fd491b3e9f19c506c01fcea8ca396)
BibTeX
@misc{dc81adca3429ce,
  title = {SITAR: A Situational Distortion Metric for Automatic Speech Recognition Evaluation — arXiv:2610.21440},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.21440},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_81adca3429ce; sha256 a5ef7d543adad872cf5f8e6405297500e11fd491b3e9f19c506c01fcea8ca396},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.21440. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In automatic speech recognition (ASR), transcription errors with similar lexical or semantic discrepancies can have markedly different consequences for the situation conveyed by speech. Capturing these differences requires not merely contextual assessment, but an understanding of what the spoken discourse describes, establishes, or seeks to bring about. However, relying on LLM judges introduces computational cost and scoring variability. We introduce SITAR, a situational distortion metric for ASR, realized through a comprehensive end-to-end construction framework. Specifically, this framework aggregates situational severity and cumulative extent into a unified score via a bounded recursive formulation. Using targets derived from this formulation with limited offline LLM support, we train SITAR as a lightweight BERT-family regressor that scores reference-hypothesis pairs in a single forward pass, entirely free of any prohibitive runtime costs. Recognizing that introducing a new metric demands rigorous justification, we formalize a principled validation protocol spanning both extrinsic validity, assessed at macro and micro levels, and intrinsic scorer reliability. The former, evaluated across diverse datasets and models, demonstrates that SITAR reliably tracks broad capability scaling while providing orthogonal diagnostic utility, whereas the latter, examined on a purpose-built controlled benchmark, confirms the operational soundness of the scoring instrument itself. Through systematic validation, we demonstrate that SITAR holds strong potential as a practical, lightweight complement to existing ASR metrics. We will release the full recipes (code, model weights, dataset, etc.) after the internal review to support community research.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.21440 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:44:47.653Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).