Metraipolis › archive › dc_fdc9ea896d1e

The Lay of the Land in Idiom-Translation Quality Estimation — arXiv:2610.35261

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.35261
License (as archived)
CC0-1.0
Archived
2026-10-27T08:12:20.639Z
SHA-256 of the archived text
889ea098b4ba783278953d134f777c59c692623af63acfc5ae6eb8387330f873
Stable URL of this page
https://metraipolis.com/documento/dc_fdc9ea896d1e

How to cite

The Lay of the Land in Idiom-Translation Quality Estimation — arXiv:2610.35261 — arXiv (en), https://arxiv.org/abs/2610.35261 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fdc9ea896d1e (Metraipolis archive dc_fdc9ea896d1e, sha256 889ea098b4ba783278953d134f777c59c692623af63acfc5ae6eb8387330f873)
BibTeX
@misc{dcfdc9ea896d1e,
  title = {The Lay of the Land in Idiom-Translation Quality Estimation — arXiv:2610.35261},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.35261},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fdc9ea896d1e; sha256 889ea098b4ba783278953d134f777c59c692623af63acfc5ae6eb8387330f873},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.35261. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Idioms have long been a challenge for machine translation, and while large language models (LLMs) have proven increasingly adept at handling them, whether quality estimation (QE) systems have kept pace and can reliably score idiomatic renderings of these expressions above faulty literal ones remains unclear. We introduce the first parallel dataset pairing idiom-bearing English sentences with both literal and idiomatic translations across nine target languages, augmented into multi-sentence segments and a graded literality continuum. Using this dataset, we benchmark eleven conventional QE metrics from the COMET and MetricX families and ten LLMs from the Qwen-3, Gemma-4 and OLMo-3 families used as GEMBA-style judges. We find that most systems, especially older metrics and smaller models, display a marked literality bias, rating literal translations of idioms above idiomatic ones. Higher parameter counts and newer training sets appear to reduce this bias to an extent; nevertheless, of all the benchmarked systems, only MetricX-24-Hybrid (XL, XXL) and Gemma-4-31B consistently rate idiomatic translations of idioms above literal ones, and only Gemma-4-31B reliably ranks intermediate degrees of idiomaticity and literality. Our findings caution against deploying standard QE metrics in domains rich in figurative language, where higher scores might be rewarding undesirable literalism.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.35261 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:56:00.260Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).