Metraipolis › archive › dc_75ae0f2242ff

Terminal-Bench-LILT: Multilingual Agentic Coding Benchmark Grounded in Language, Region, and Culture — arXiv:2608.28641

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.28641
License (as archived)
CC0-1.0
Archived
2026-10-17T07:47:10.806Z
SHA-256 of the archived text
b339b22a56bd1bba259da5f0ba52c89c1f699a2191d0a05d3225f5fbfa699e48
Stable URL of this page
https://metraipolis.com/documento/dc_75ae0f2242ff

How to cite

Terminal-Bench-LILT: Multilingual Agentic Coding Benchmark Grounded in Language, Region, and Culture — arXiv:2608.28641 — arXiv (en), https://arxiv.org/abs/2608.28641 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_75ae0f2242ff (Metraipolis archive dc_75ae0f2242ff, sha256 b339b22a56bd1bba259da5f0ba52c89c1f699a2191d0a05d3225f5fbfa699e48)
BibTeX
@misc{dc75ae0f2242ff,
  title = {Terminal-Bench-LILT: Multilingual Agentic Coding Benchmark Grounded in Language, Region, and Culture — arXiv:2608.28641},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.28641},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_75ae0f2242ff; sha256 b339b22a56bd1bba259da5f0ba52c89c1f699a2191d0a05d3225f5fbfa699e48},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.28641. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Most evaluations for coding agents are conducted exclusively in English, which does not reflect real-world multilingual deployment. We present Terminal-Bench-LILT, a suite of 300 authentic coding tasks in ten languages: Arabic, Czech, German, Spanish, Hindi, Japanese, Korean, Serbian, Turkish, and Chinese. Each task targets issues specific to non-English software development that have no direct English equivalent, e.g., internationalization, encoding, text normalization, and cultural conventions. All tasks are authored by native-speaker programmers and validated through a multi-stage quality control pipeline. Evaluation of six frontier models reveals that even the strongest model reaches only 63.1\% pass rate, with many tasks unsolved by any model. Performance varies substantially by language and does not track general coding benchmark rankings, highlighting that multilingual coding competence is a distinct and underexplored capability axis. Sample tasks are available at this https URL

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.28641 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T07:28:26.672Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).