Metraipolis › archive › dc_73ef074efbbd

Benchmarking the Titans: A Multi-Dimensional Empirical Evaluation of LLM Code Generation Quality in the .NET Ecosystem — arXiv:2608.22529

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.22529
License (as archived): CC0-1.0
Archived: 2026-10-18T08:12:37.551Z
SHA-256 of the archived text: 0040106c2219074ff047da7a137be85ae7892a1de6c564dd2e1472242d168e20
Stable URL of this page: https://metraipolis.com/documento/dc_73ef074efbbd

How to cite

Benchmarking the Titans: A Multi-Dimensional Empirical Evaluation of LLM Code Generation Quality in the .NET Ecosystem — arXiv:2608.22529 — arXiv (en), https://arxiv.org/abs/2608.22529 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_73ef074efbbd (Metraipolis archive dc_73ef074efbbd, sha256 0040106c2219074ff047da7a137be85ae7892a1de6c564dd2e1472242d168e20)

BibTeX

@misc{dc73ef074efbbd,
  title = {Benchmarking the Titans: A Multi-Dimensional Empirical Evaluation of LLM Code Generation Quality in the .NET Ecosystem — arXiv:2608.22529},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.22529},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_73ef074efbbd; sha256 0040106c2219074ff047da7a137be85ae7892a1de6c564dd2e1472242d168e20},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.22529. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluating Large Language Model (LLM) code generation quality requires examining not just whether the generated code is correct, but whether it is maintainable, efficient, and stylistically sound, all of which are qualities of direct importance to software engineering practitioners. Existing benchmarks reduce evaluation to a single Pass@k metric, which obscures critical trade-offs between functional correctness and structural quality. A further limitation is the near-exclusive focus on Python, leaving enterprise-relevant ecosystems such as C# and .NET without dedicated evaluation. This paper presents an automated, multi-dimensional evaluation framework for C# code generation, applying it to four state-of-the-art LLMs: GPT, Gemini, Claude, and Grok. We conduct a controlled experiment across 85 algorithmic tasks derived from HumanEval, generating and evaluating 340 solutions in total, in which each solution is assessed across three independent dimensions: functional correctness via automated unit testing, static code quality via Roslyn AST analysis, and runtime efficiency via adversarial BenchmarkDotNet profiling. Our central finding reveals a substantial gap between correctness and quality attributes (Pearson r = 0.075), demonstrating that Pass@k rankings systematically misrepresent the full LLM performance profile in software engineering contexts. We further characterize GPT's bimodal failure behavior.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-25 in cs.SE).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2608.22529 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T08:07:33.003Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-08-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).