Metraipolis › archive › dc_bf4b975e01d1

Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem — arXiv:2608.19626

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.19626
License (as archived): CC0-1.0
Archived: 2026-10-18T11:49:45.765Z
SHA-256 of the archived text: 7929a41e88852bce78427c89d812b418bf6782af4fcc8e5560cf3521704e6854
Stable URL of this page: https://metraipolis.com/documento/dc_bf4b975e01d1

How to cite

Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem — arXiv:2608.19626 — arXiv (en), https://arxiv.org/abs/2608.19626 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf4b975e01d1 (Metraipolis archive dc_bf4b975e01d1, sha256 7929a41e88852bce78427c89d812b418bf6782af4fcc8e5560cf3521704e6854)

BibTeX

@misc{dcbf4b975e01d1,
  title = {Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem — arXiv:2608.19626},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.19626},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf4b975e01d1; sha256 7929a41e88852bce78427c89d812b418bf6782af4fcc8e5560cf3521704e6854},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.19626. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Execution feedback is often treated as a self-verifying signal for improving LLM-generated tests. However, when generated inputs are executed on a single accepted program and its outputs are used as ground truth, invalid or underspecified inputs can create spurious fault detections and apparent evolutionary gains. We audit this failure mode in feedback-driven test generation using 142 development tasks, 114 locked external tasks, and 138 held-out tasks, with two code models, three seeds, and fault-cross-fitted real submissions. On external inputs for which three accepted implementations agree, generated outputs match the panel on only 27.79% and 50.12% of cases. A single-reference oracle inflates the measured gain from evolution by 9.46-14.85 percentage points; after auditing, equal-budget independent resampling outperforms mutation-based evolution by 6.01-18.83 points. We further compare a genuine three-round feedback loop with a density-matched placebo. External Real-Placebo differences are +0.13 and -0.50 points, while held-out differences are +1.99 and +0.28 points and do not provide robust evidence of fine-grained feedback benefit. A blinded semantic audit by two software engineering doctoral students classifies 94.41% of panel-disconfirmed inputs as invalid but 3.60% as valid, showing that panel disagreement is informative but not semantic proof. We propose an audit-and-placebo protocol that separates verifier artifacts, interaction scaffolding, and grounded feedback credit in evaluations of self-evolving test generators.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-21 in cs.SE).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2608.19626 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T11:48:10.185Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-08-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).