Metraipolis › archive › dc_3d24dc7ff356
Making COMET Comparable Across Scripts: Diagnosis and Correction of Tokeniser-Induced Script Bias in Indic MT Evaluation — arXiv:2610.08159
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.08159
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T11:23:21.108Z
- SHA-256 of the archived text
cadd615610dfd4ed7d0cbfb96f49b7a1111a38a8c49b0a03e6d58fd3283cce61- Stable URL of this page
https://metraipolis.com/documento/dc_3d24dc7ff356
How to cite
Making COMET Comparable Across Scripts: Diagnosis and Correction of Tokeniser-Induced Script Bias in Indic MT Evaluation — arXiv:2610.08159 — arXiv (en), https://arxiv.org/abs/2610.08159 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3d24dc7ff356 (Metraipolis archive dc_3d24dc7ff356, sha256 cadd615610dfd4ed7d0cbfb96f49b7a1111a38a8c49b0a03e6d58fd3283cce61)
BibTeX
@misc{dc3d24dc7ff356,
title = {Making COMET Comparable Across Scripts: Diagnosis and Correction of Tokeniser-Induced Script Bias in Indic MT Evaluation — arXiv:2610.08159},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.08159},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3d24dc7ff356; sha256 cadd615610dfd4ed7d0cbfb96f49b7a1111a38a8c49b0a03e6d58fd3283cce61},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08159. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
COMET reports translation quality as a single number, and that number is routinely compared across target languages written in different scripts. Such a comparison assumes Script Invariance: the score should not depend on the writing system that carries the target. We test it on IndicMT Eval by re-encoding the target into Latin script, which changes orthographic form while holding content and human ratings fixed. Script identity then accounts for 22.9% of native-script COMET variance, and agreement with annotators falls in all five languages studied. We trace the effect to the tokeniser and measure it with three label-free diagnostics. The bias is two faults, not one. Scores from different scripts occupy incompatible ranges, and within a single script the metric orders translations less accurately. No order-preserving transform of the score can repair the second fault. The first is removed exactly by COMET-QN, which maps the score distribution of each (language, script) pair onto a shared reference. Pooled agreement with annotators rises from 0.300 to 0.399, which is what makes scores from different scripts safe to place on one axis, and every within-language ordering is provably preserved. A regressor over parity features recovers a further 17.1% of the lost sensitivity. The remainder belongs to the encoder, and no post-processing can reach it. We therefore recommend publishing the normalised score, the three diagnostics, and the identity of the tokeniser they were computed against, so that a reader can tell how much of a score reflects translation quality and how much reflects the writing system. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.08159 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T11:15:39.449Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).