Metraipolis › archive › dc_096914ffd0eb
M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models — arXiv:2608.03803
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.03803
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T12:31:11.187Z
- SHA-256 of the archived text
d353bd9507faf746ed985c1509abfcd7a0470cf0265d360a39ef0dd94ae0f9b6- Stable URL of this page
https://metraipolis.com/documento/dc_096914ffd0eb
How to cite
M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models — arXiv:2608.03803 — arXiv (en), https://arxiv.org/abs/2608.03803 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_096914ffd0eb (Metraipolis archive dc_096914ffd0eb, sha256 d353bd9507faf746ed985c1509abfcd7a0470cf0265d360a39ef0dd94ae0f9b6)
BibTeX
@misc{dc096914ffd0eb,
title = {M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models — arXiv:2608.03803},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.03803},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_096914ffd0eb; sha256 d353bd9507faf746ed985c1509abfcd7a0470cf0265d360a39ef0dd94ae0f9b6},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03803. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency. We introduce M-GATE (Multilingual Grammar, Accuracy in Translation, and Efficiency), a benchmark of linguistic proficiency spanning 30 typologically diverse languages from high- to low-resource. M-GATE comprises three tasks: grammatical error detection on linguist-crafted, adversarially selected sentences that turn on hard, language-specific phenomena; round-trip translation of shared English sources across 29 target languages, scored by a three-provider LLM judge panel validated against professional annotators; and a supplementary tokenizer-efficiency measure. We evaluate over 50 models in more than 80 configurations. Fluency and proficiency come apart sharply: models that translate competently sit near chance on the adversarial grammar items, the best reaching a Matthews correlation coefficient (MCC) of only 0.36, and their errors lean systematically toward under-flagging, accepting ungrammatical text rather than raising false alarms. Translation quality closely tracks a language's share of pretraining data (r = 0.86 against log Common Crawl share), producing a steep low-resource penalty that is nonetheless narrowing with successive model releases. Enabling reasoning reliably improves translation, while its effect on error detection is smaller and for some models negative, so the best configuration is task-dependent. To resist contamination, test items are kept private behind a continuously updated public leaderboard, with illustrative examples released ( this https URL ). Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.03803 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T12:17:37.780Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).