Metraipolis › archive › dc_be1ecce1177a
TestPrism: Rethinking Test Evaluation Beyond a Single Reference — arXiv:2610.12289
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.12289
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T06:26:44.141Z
- SHA-256 of the archived text
581994653bbe7e38f8923aa92962173cb9085267ae72cff5723d61303a30b064- Stable URL of this page
https://metraipolis.com/documento/dc_be1ecce1177a
How to cite
TestPrism: Rethinking Test Evaluation Beyond a Single Reference — arXiv:2610.12289 — arXiv (en), https://arxiv.org/abs/2610.12289 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_be1ecce1177a (Metraipolis archive dc_be1ecce1177a, sha256 581994653bbe7e38f8923aa92962173cb9085267ae72cff5723d61303a30b064)
BibTeX
@misc{dcbe1ecce1177a,
title = {TestPrism: Rethinking Test Evaluation Beyond a Single Reference — arXiv:2610.12289},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.12289},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_be1ecce1177a; sha256 581994653bbe7e38f8923aa92962173cb9085267ae72cff5723d61303a30b064},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.12289. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language model (LLM) coding agents have advanced test generation across diverse programming tasks. However, the common practice of evaluating tests against a single reference solution overlooks alternative valid implementations and can overstate test quality. We introduce TestPrism, comprising 300 test tasks from 17 sources and 3000 candidate implementations, evenly split between valid and invalid solutions. Its primary metric, Joint Success Function, requires the generated tests to fail on the initial program state, accept every valid candidate, and reject every invalid candidate. Across fourteen baseline coding agent configurations, Joint Success Function reaches only 28.00%, whereas single reference success reaches 59.67%. Our analysis reveals missed behaviors, unsupported assertions, and faulty test construction. To address these weaknesses, we introduce TestHelix, which combines heterogeneous synthesis of test and repair pairs with peer cross validation and recursive self improvement (RSI). Across two models, TestHelix improves Joint Success Function by 8.67 to 9.00 percentage points over the native harness comparators in the TestHelix evaluation Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2610.12289 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T05:50:45.989Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).