Metraipolis › archive › dc_3cdc56b65905
VibeCheck: Assessing the Quality of LLM-Generated Unit Tests: A Multi-agent Empirical Study across Heterogeneous Repositories — arXiv:2609.05978
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.05978
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T14:09:00.461Z
- SHA-256 of the archived text
251b5a7028d7091632218989c6324d7cbef6af1413bbf60253e25ac4eeea32c0- Stable URL of this page
https://metraipolis.com/documento/dc_3cdc56b65905
How to cite
VibeCheck: Assessing the Quality of LLM-Generated Unit Tests: A Multi-agent Empirical Study across Heterogeneous Repositories — arXiv:2609.05978 — arXiv (en), https://arxiv.org/abs/2609.05978 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3cdc56b65905 (Metraipolis archive dc_3cdc56b65905, sha256 251b5a7028d7091632218989c6324d7cbef6af1413bbf60253e25ac4eeea32c0)
BibTeX
@misc{dc3cdc56b65905,
title = {VibeCheck: Assessing the Quality of LLM-Generated Unit Tests: A Multi-agent Empirical Study across Heterogeneous Repositories — arXiv:2609.05978},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.05978},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3cdc56b65905; sha256 251b5a7028d7091632218989c6324d7cbef6af1413bbf60253e25ac4eeea32c0},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.05978. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLM-based IDE agents are increasingly used to generate repository-grounded unit tests, yet common evaluations often rely on execution success or coverage. These metrics can miss deeper quality issues such as weak assertions, missing edge cases, poor isolation, and limited maintainability. This paper presents VibeCheck, an empirical study of unit test generation across 15 student-developed Python and JavaScript/TypeScript repositories. We evaluate Kiro, Antigravity, and Cursor with Claude Sonnet 4.5 as the underlying agent, under repository-only, zero-shot conditions using a five-dimensional rubric covering runnability, assertion strength, logic and edge-case coverage, isolation/determinism, and maintainability. We also apply leave-one-out cross-agent peer evaluation to compare tools and identify failure patterns. Results show a clear execution-adequacy gap: generated tests are often runnable but frequently lack strong assertions and meaningful behavioral coverage. Weak assertions and missing edge cases occur more often than blocking failures, showing that runnable tests can still be shallow. VibeCheck provides a reliability-oriented framework for evaluating LLM-generated tests beyond pass/fail outcomes. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-09 in cs.SE). Categorie arXiv: Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2609.05978 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T13:58:36.350Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).