Metraipolis › archive › dc_2bea1c801923
Evaluating Language Models on Cross-Language Code Functional Equivalence — arXiv:2608.23961
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.23961
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T06:20:45.776Z
- SHA-256 of the archived text
0a6ecdc73db36f70e7ca57c6085d45729a2bed2891b965b21fece848f3cd2a1f- Stable URL of this page
https://metraipolis.com/documento/dc_2bea1c801923
How to cite
Evaluating Language Models on Cross-Language Code Functional Equivalence — arXiv:2608.23961 — arXiv (en), https://arxiv.org/abs/2608.23961 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2bea1c801923 (Metraipolis archive dc_2bea1c801923, sha256 0a6ecdc73db36f70e7ca57c6085d45729a2bed2891b965b21fece848f3cd2a1f)
BibTeX
@misc{dc2bea1c801923,
title = {Evaluating Language Models on Cross-Language Code Functional Equivalence — arXiv:2608.23961},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.23961},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2bea1c801923; sha256 0a6ecdc73db36f70e7ca57c6085d45729a2bed2891b965b21fece848f3cd2a1f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.23961. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Background: Large Language Models (LLMs) have demonstrated strong performance across a variety of code-understanding tasks, leading many to believe that they can reason about program semantics. However, existing evaluations primarily focus on single-language settings or rely on synthetically generated code, raising concerns about whether current results reflect true semantic understanding. Aims: We investigate whether LLMs can accurately judge functional equivalence across different programming languages in human-written code, a setting that requires deeper reasoning beyond superficial similarity. Method: We introduce PolyHuman, a dataset of human-written programs in CPP, Java, and Python. Using this dataset, we evaluate intra- and inter-language equivalence detection across open-weight and proprietary LLMs, selecting GPT-o4-mini as a representative model to assess stability. We then manually analyze 81 cases of systematic disagreement in which models incorrectly judge functional equivalence, examining the code logic and the generated Chain-of-Thought reasoning. Finally, we categorize these failures and compare them across GPT-o4-mini, Claude-Opus-4.7, and Gemini-3-Flash to determine whether they reflect model-specific issues or broader limitations of state-of-the-art LLMs. Results: We identify a difficulty-dependent breakdown in equivalence judgment (harder problems make the model increasingly prone to misclassifying non-equivalent code as equivalent), a model-specific sensitivity to programming language for the best-performing model (particularly a more conservative behavior on Python), and a partial reliance on similarity-based cues. GPT-o4-mini also shows substantial run-to-run instability under identical settings, indicating inconsistent rather than absent capability. Conclusions: Current LLMs do not reliably capture functional equivalence within or across languages. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-26 in cs.CL). Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.23961 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T06:17:29.399Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-26?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).