Metraipolis › archive › dc_42b5c6fb2c19

Measuring the Cost of Variety Conflation in Multilingual MT Evaluation: Adding Mozambican Xichangana, Nyanja and Sena to FLORES+ — arXiv:2609.13847

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.13847
License (as archived): CC0-1.0
Archived: 2026-10-15T23:14:37.269Z
SHA-256 of the archived text: 28edda72a5936ed48d38da65fde2fc07756be4a33326e6b566551e757b0cd951
Stable URL of this page: https://metraipolis.com/documento/dc_42b5c6fb2c19

How to cite

Measuring the Cost of Variety Conflation in Multilingual MT Evaluation: Adding Mozambican Xichangana, Nyanja and Sena to FLORES+ — arXiv:2609.13847 — arXiv (en), https://arxiv.org/abs/2609.13847 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_42b5c6fb2c19 (Metraipolis archive dc_42b5c6fb2c19, sha256 28edda72a5936ed48d38da65fde2fc07756be4a33326e6b566551e757b0cd951)

BibTeX

@misc{dc42b5c6fb2c19,
  title = {Measuring the Cost of Variety Conflation in Multilingual MT Evaluation: Adding Mozambican Xichangana, Nyanja and Sena to FLORES+ — arXiv:2609.13847},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.13847},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_42b5c6fb2c19; sha256 28edda72a5936ed48d38da65fde2fc07756be4a33326e6b566551e757b0cd951},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.13847. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In this paper, we extend FLORES+ with Portuguese-source evaluation sets for three Mozambican Bantu varieties: Xichangana, Mozambican Nyanja, and Sena. We compare Xichangana with the existing Tsonga reference and Mozambican Nyanja with Chichewa, and evaluate NLLB-200, Google Translate, GPT, and a variant-aware NLLB model. Holding system output fixed reveals substantial reference sensitivity. On \textit{devtest}, changing only the reference from Tsonga to Xichangana reduces spBLEU by 13.10 points for NLLB-200 and 15.30 for Google. On matched Nyanja subsets, replacing Chichewa with Mozambican Nyanja produces smaller but consistent reductions of 3.03 and 6.10 spBLEU, respectively. Variant-aware fine-tuning reverses this pattern on the intended targets: relative to NLLB-200, it improves Xichangana by 7.04 spBLEU and Mozambican Nyanja by 5.33 on \textit{devtest}, while losing performance on the sibling references. GPT is competitive on Tsonga and Chichewa but substantially weaker on the Mozambican varieties. For Sena, the finetuned model reaches 12.64 spBLEU and 36.21 chrF++ on \textit{devtest}. These findings motivate variety-aware language identifiers, references, and reporting for cross-border languages or language dialects/variants. The data is publicly available on Hugging Face at this https URL

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.13847 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T22:51:09.845Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).