Metraipolis › archive › dc_3f08eeaea305

A Study on the Impact of Natural Language Differences in Prompts on Automatic Code Generation Using LLMs — arXiv:2609.18311

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.18311
License (as archived)
CC0-1.0
Archived
2026-10-15T15:38:48.263Z
SHA-256 of the archived text
5a818b38992e1ec60c259eb016b054ceac5498d12dd7832b93ac3b193033ced3
Stable URL of this page
https://metraipolis.com/documento/dc_3f08eeaea305

How to cite

A Study on the Impact of Natural Language Differences in Prompts on Automatic Code Generation Using LLMs — arXiv:2609.18311 — arXiv (en), https://arxiv.org/abs/2609.18311 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3f08eeaea305 (Metraipolis archive dc_3f08eeaea305, sha256 5a818b38992e1ec60c259eb016b054ceac5498d12dd7832b93ac3b193033ced3)
BibTeX
@misc{dc3f08eeaea305,
  title = {A Study on the Impact of Natural Language Differences in Prompts on Automatic Code Generation Using LLMs — arXiv:2609.18311},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.18311},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3f08eeaea305; sha256 5a818b38992e1ec60c259eb016b054ceac5498d12dd7832b93ac3b193033ced3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.18311. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large Language Models (LLMs) have demonstrated remarkable performance in automatic code generation tasks, thereby encouraging new research in this area. Although numerous studies have explored LLM-based code generation, the impact of the natural language in input prompts remains unexplored (language bias). This study aims to (1) quantify how the natural language of input prompts influences LLM-based code generation performance and (2) evaluate a mitigation strategy to reduce language bias in code generation. We assess code generation Accuracy on AtCoder, LeetCode, and BigCodeBench. To quantify the language bias on code generation, each problem is presented in English, Japanese, and Chinese. We use seven LLMs (GPT-4o, o3-mini, DeepSeek-V3.2, Llama-3, Qwen2.5-Coder-14B, Qwen2.5-Coder-0.5B, and GitHub Copilot) and assess their performance in terms of Accuracy (the number of problems for which generated code passes all test cases). We compare Accuracy before and after translation to evaluate the effectiveness of translation as a mitigation strategy. We observed that the natural language of problem statements affects LLM-based code generation performance. Specifically, the languages officially supported by each dataset achieved the highest median Accuracy. Also, translation improved Accuracy, but its effectiveness was not consistent across datasets and model types. We found that AtCoder contained a particularly high proportion of narrative-style problem statements and longer problem statements. Natural language significantly affects LLM code generation accuracy. Translation can mitigate language bias in some settings, but its effectiveness depends on the dataset and model type. Furthermore, the narrative aspects and context length of input prompts are important factors related to language bias and the effectiveness of translation as a mitigation strategy.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-17 in cs.SE).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2609.18311 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T15:04:30.923Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).