Metraipolis › archive › dc_170278fca248

Measuring the Cross-Lingual Comprehension Gap: How the language of the evidence shapes what language models understand — arXiv:2608.06506

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.06506
License (as archived)
CC0-1.0
Archived
2026-10-19T09:46:42.380Z
SHA-256 of the archived text
4cb740483979781615d807e5f2cd29ac0d8e7516dec06a67a81d7291fec4e7e4
Stable URL of this page
https://metraipolis.com/documento/dc_170278fca248

How to cite

Measuring the Cross-Lingual Comprehension Gap: How the language of the evidence shapes what language models understand — arXiv:2608.06506 — arXiv (en), https://arxiv.org/abs/2608.06506 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_170278fca248 (Metraipolis archive dc_170278fca248, sha256 4cb740483979781615d807e5f2cd29ac0d8e7516dec06a67a81d7291fec4e7e4)
BibTeX
@misc{dc170278fca248,
  title = {Measuring the Cross-Lingual Comprehension Gap: How the language of the evidence shapes what language models understand — arXiv:2608.06506},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.06506},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_170278fca248; sha256 4cb740483979781615d807e5f2cd29ac0d8e7516dec06a67a81d7291fec4e7e4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.06506. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Language models are often evaluated as though capabilities demonstrated in English remain equally available when the same content is presented in other languages. Traditional multilingual benchmarks rarely isolate language while holding content, question, reference answer, model, and evaluation unit constant. We define the Cross-Lingual Comprehension Gap (CLCG) as the reduction in response quality when the same content and question are presented in a target language rather than in English. Using ParallelQA-18, a professionally human-translated parallel corpus, we evaluate five models from five laboratories on a stratified sample of 150 articles across 18 languages (English reference; Portuguese high-resource baseline; 16 targets spanning Joshi et al. 2020 classes 0-4). A within-item design varies only passage language. The primary estimator contrasts English versus pooled target-language Token-F1 micro-means on higher-complexity open-ended questions, with article-cluster bootstrap intervals. The primary pooled CLCG is 0.078 (95% CI 0.072-0.084), about a 17% reduction relative to the English score; the equal-language macro summary is 0.077. Net of Portuguese, the macro gap is 0.016 (95% CI 0.013-0.020). Language-level CLCG is negatively associated with Joshi resource class (rho = -0.594, p = 0.015, n = 16). In blinded paired human evaluations, higher-resource responses are preferred in 61.6% of decisive judgments (estimated preference probability 0.655, 95% CI 0.558-0.741). Capabilities shown in English should not be assumed to transfer equally to other languages; English-centered evaluations may overestimate quality for users of low-resource languages.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-10 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.06506 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T09:46:37.194Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).