Metraipolis › archive › dc_16e2d6f032c6
Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks — arXiv:2609.03734
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.03734
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T01:12:04.670Z
- SHA-256 of the archived text
83acef45f73ec4fdc199dbbe426ebd63ba9cf1325435aa018bb1f3a7cfb56891- Stable URL of this page
https://metraipolis.com/documento/dc_16e2d6f032c6
How to cite
Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks — arXiv:2609.03734 — arXiv (en), https://arxiv.org/abs/2609.03734 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_16e2d6f032c6 (Metraipolis archive dc_16e2d6f032c6, sha256 83acef45f73ec4fdc199dbbe426ebd63ba9cf1325435aa018bb1f3a7cfb56891)
BibTeX
@misc{dc16e2d6f032c6,
title = {Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks — arXiv:2609.03734},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.03734},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_16e2d6f032c6; sha256 83acef45f73ec4fdc199dbbe426ebd63ba9cf1325435aa018bb1f3a7cfb56891},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03734. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
BLEU-4 is the standard metric for evaluating sign language translation (SLT), but spoken-language metrics may not adequately reflect sign language proficiency. The multimodal, low-resource context of SLT allows models to exploit spurious correlations and spoken-language priors, rather than learning stronger sign representations. In this paper, we evaluate the relationship between spatio-temporal understanding and BLEU-4 across six SLT models on Phoenix-2014T and CSL-Daily, showing that gains in BLEU-4 are not on their own evidence of better sign language understanding. This work introduces an alternative inspired by language-learning assessment, using an open-weight-LLM QA protocol that measures salient content preservation. It aligns more closely with human rankings and is six to seven times more paraphrase-invariant than BLEU-4. Applied to SLT, this protocol targets content transfer, is more robust to train-test overlap, and gives a different picture of the field: the five gloss-free systems are largely within noise of one another on Phoenix-2014T, while the gloss-supervised system stands 9.3 points higher, a gap invisible to BLEU-4. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.03734 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T00:28:02.998Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).