Metraipolis › archive › dc_3e764eb28301

LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs — arXiv:2608.06041

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.06041
License (as archived)
CC0-1.0
Archived
2026-10-27T08:04:38.687Z
SHA-256 of the archived text
acf00d74c77995bdcaaad31c9c3d6a1c805731bdec3782bd68f72698b7e56f26
Stable URL of this page
https://metraipolis.com/documento/dc_3e764eb28301

How to cite

LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs — arXiv:2608.06041 — arXiv (en), https://arxiv.org/abs/2608.06041 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3e764eb28301 (Metraipolis archive dc_3e764eb28301, sha256 acf00d74c77995bdcaaad31c9c3d6a1c805731bdec3782bd68f72698b7e56f26)
BibTeX
@misc{dc3e764eb28301,
  title = {LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs — arXiv:2608.06041},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.06041},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3e764eb28301; sha256 acf00d74c77995bdcaaad31c9c3d6a1c805731bdec3782bd68f72698b7e56f26},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.06041. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) have been shown to exhibit strong Python preferences when generating project-level code, but there is currently no systematic way to measure this behaviour across new models. To bridge this gap, we introduce LangChoiceBench, a project-level code-generation benchmark for measuring Python preference, recommendation-implementation consistency, and language diversity. LangChoiceBench covers 28 projects across seven software areas where Python is often a poor default. We evaluate 25 diverse LLMs and find that Python remains heavily over-selected, recommendation-implementation consistency is low, and smaller open-weight models generally show stronger Python preference and lower language diversity. We further analyse 9,826 reasoning traces and find that most Python choices are automatic or driven primarily by ease, rather than explicit consideration of project requirements. In a smaller but important set of cases, models fabricate contextual support for choosing Python - a failure mode we call phantom evidence - or produce code that contradicts the language selected in their own reasoning.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-07 in cs.SE).
Categorie arXiv: Software Engineering (cs.SE) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.06041 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:55:47.048Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-08-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).