Metraipolis › archive › dc_16e2d6f032c6

Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks — arXiv:2609.03734

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.03734
License (as archived)
CC0-1.0
Archived
2026-10-17T01:12:04.670Z
SHA-256 of the archived text
83acef45f73ec4fdc199dbbe426ebd63ba9cf1325435aa018bb1f3a7cfb56891
Stable URL of this page
https://metraipolis.com/documento/dc_16e2d6f032c6

How to cite

Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks — arXiv:2609.03734 — arXiv (en), https://arxiv.org/abs/2609.03734 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_16e2d6f032c6 (Metraipolis archive dc_16e2d6f032c6, sha256 83acef45f73ec4fdc199dbbe426ebd63ba9cf1325435aa018bb1f3a7cfb56891)
BibTeX
@misc{dc16e2d6f032c6,
  title = {Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks — arXiv:2609.03734},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.03734},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_16e2d6f032c6; sha256 83acef45f73ec4fdc199dbbe426ebd63ba9cf1325435aa018bb1f3a7cfb56891},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03734. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

BLEU-4 is the standard metric for evaluating sign language translation (SLT), but spoken-language metrics may not adequately reflect sign language proficiency. The multimodal, low-resource context of SLT allows models to exploit spurious correlations and spoken-language priors, rather than learning stronger sign representations. In this paper, we evaluate the relationship between spatio-temporal understanding and BLEU-4 across six SLT models on Phoenix-2014T and CSL-Daily, showing that gains in BLEU-4 are not on their own evidence of better sign language understanding. This work introduces an alternative inspired by language-learning assessment, using an open-weight-LLM QA protocol that measures salient content preservation. It aligns more closely with human rankings and is six to seven times more paraphrase-invariant than BLEU-4. Applied to SLT, this protocol targets content transfer, is more robust to train-test overlap, and gives a different picture of the field: the five gloss-free systems are largely within noise of one another on Phoenix-2014T, while the gloss-supervised system stands 9.3 points higher, a gap invisible to BLEU-4.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.03734 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T00:28:02.998Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).