Metraipolis › archive › dc_44baa7251eda
Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER — arXiv:2609.21663
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.21663
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-10T01:27:35.445Z
- SHA-256 of the archived text
11252b6e23b1555cade89fd39254d609d2a613cdc78fb154c8b9cbc9b1261676- Stable URL of this page
https://metraipolis.com/documento/dc_44baa7251eda
How to cite
Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER — arXiv:2609.21663 — arXiv (en), https://arxiv.org/abs/2609.21663 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_44baa7251eda (Metraipolis archive dc_44baa7251eda, sha256 11252b6e23b1555cade89fd39254d609d2a613cdc78fb154c8b9cbc9b1261676)
BibTeX
@misc{dc44baa7251eda,
title = {Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER — arXiv:2609.21663},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.21663},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_44baa7251eda; sha256 11252b6e23b1555cade89fd39254d609d2a613cdc78fb154c8b9cbc9b1261676},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21663. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equally costly, regardless of whether it changes meaning. This raises the question: does WER actually track how humans judge ASR transcript quality? We introduce HATS-en, an English dataset for human-centered ASR evaluation. Using this dataset, we benchmark lexical metrics against several configurations of BERTScore and SemDist, varying the language model, layer, and pooling strategy. We find that WER agrees least with human judgment among all metrics tested, that the best-performing SemDist configurations achieve the highest overall agreement, ahead of CER and BERTScore, and that no single model is best across settings. CER, despite its simplicity and low cost, remains remarkably close to these best configurations. In line with prior recommendations, our results support shifting ASR evaluation toward CER both for English and for morphosyllabic writing systems as it is a more interpretable and low-cost metric for what evaluation should actually capture, and using SemDist as a complementary evaluation. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.21663 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-10T01:00:30.293Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).