Metraipolis › archive › dc_3be19645979c

CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation — arXiv:2608.03535

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.03535
License (as archived)
CC0-1.0
Archived
2026-10-31T12:43:28.459Z
SHA-256 of the archived text
31a813981a0dcb0a768f7404ba4b0116518da4f8f7212bf99e8601b65bfe9ef1
Stable URL of this page
https://metraipolis.com/documento/dc_3be19645979c

How to cite

CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation — arXiv:2608.03535 — arXiv (en), https://arxiv.org/abs/2608.03535 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3be19645979c (Metraipolis archive dc_3be19645979c, sha256 31a813981a0dcb0a768f7404ba4b0116518da4f8f7212bf99e8601b65bfe9ef1)
BibTeX
@misc{dc3be19645979c,
  title = {CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation — arXiv:2608.03535},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.03535},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3be19645979c; sha256 31a813981a0dcb0a768f7404ba4b0116518da4f8f7212bf99e8601b65bfe9ef1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03535. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large Language Models are increasingly evaluated for code generation using test-based benchmarks. The validity of such evaluations depends on the reliability of their references and tests, while test-based correctness captures only part of the observable properties of generated code. We present CodeAssay, a taxonomy-first benchmark of 185 Python tasks across ten software-engineering categories. It combines audited ground truth, public tests for generation and repair, hidden tests for grading, mutation-based test-suite validation, and selected code-property measures. Regrading fixed model outputs after the audit changed 170 of 1,890 correctness labels (9.0%) and increased the measured best-to-worst model spread from 11.9 to 23.7 percentage points, although aggregate correctness remained nearly unchanged. The complete and hidden test suites achieved mutation scores of 82.6% and 74.8%, respectively. Across seven proprietary LLMs, standard-prompt correctness ranged from 77.3% to 98.9%, with significant differences in 12 of 21 model pairs. On the 120 tasks solved by all 14 model-prompt configurations, no model performed best across all selected code properties. A security-focused prompt produced no significant change in correctness or consistent reduction in the selected static-analysis findings, while increasing program length and cyclomatic complexity across all models. These findings show that reliable evaluation of LLM-generated code requires validated ground truth, protected tests, and multiple explicitly interpreted measures. CodeAssay provides a reproducible basis for evidence-based model evaluation in AI-augmented software development.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-05 in cs.SE).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2608.03535 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:33:04.220Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).