Metraipolis › archive › dc_a3219bb8d11d

An Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming Tasks — arXiv:2609.18052

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.18052
License (as archived)
CC0-1.0
Archived
2026-10-15T15:38:48.248Z
SHA-256 of the archived text
45678d16fe4189ce5667f0ce03ea6209103ce22bdbbec470c32b4bc9d808dc15
Stable URL of this page
https://metraipolis.com/documento/dc_a3219bb8d11d

How to cite

An Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming Tasks — arXiv:2609.18052 — arXiv (en), https://arxiv.org/abs/2609.18052 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a3219bb8d11d (Metraipolis archive dc_a3219bb8d11d, sha256 45678d16fe4189ce5667f0ce03ea6209103ce22bdbbec470c32b4bc9d808dc15)
BibTeX
@misc{dca3219bb8d11d,
  title = {An Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming Tasks — arXiv:2609.18052},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.18052},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a3219bb8d11d; sha256 45678d16fe4189ce5667f0ce03ea6209103ce22bdbbec470c32b4bc9d808dc15},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.18052. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

This study empirically evaluates whether cost-efficient Large Language Models (LLMs) can be trusted to generate enterprise code to a written specification. Three models (Gemini Flash 3, GPT-5.4 mini and Claude Haiku 4.5) were asked to solve 992 algorithmic problems as Java Spring Boot service methods conforming to a mandated signature and data-transfer-object specification, crossing four model and agentic coding tool combinations with two prompt variants to yield eight configurations, with iteration forbidden and hardcoded answers explicitly prohibited. Eight problem statements were withheld to probe how models respond to missing input. The 7,593 resulting methods were classified by an eight-class outcome taxonomy describing what each does about producing an answer, then deployed and executed, giving 7,936 measured requests joined to that classification. Structural conformance approached ceiling, yet 38.4% of methods do not compute the value they returned and only 12.9% of returned answers were correct. Conditioning on outcome class shows that response reliability and correctness are inversely related, whereas genuinely computing methods answered least often and were correct 19.3%. Limitations include single generation runs per configuration, partial harness coverage, single-pass timing, syntactic classification, and probable corpus contamination.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-17 in cs.SE).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.18052 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T15:04:30.920Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).