Metraipolis › archive › dc_fe726914529b
Security Tests as Executable Specifications for LLM Code Generation: Benefits, Trade-offs, and Coverage Limits — arXiv:2608.09740
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.09740
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T09:01:02.503Z
- SHA-256 of the archived text
45462a11c658056f53ecdac2e5ccd067e1769e5eec8e48bbfabd7bf4beb1b892- Stable URL of this page
https://metraipolis.com/documento/dc_fe726914529b
How to cite
Security Tests as Executable Specifications for LLM Code Generation: Benefits, Trade-offs, and Coverage Limits — arXiv:2608.09740 — arXiv (en), https://arxiv.org/abs/2608.09740 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fe726914529b (Metraipolis archive dc_fe726914529b, sha256 45462a11c658056f53ecdac2e5ccd067e1769e5eec8e48bbfabd7bf4beb1b892)
BibTeX
@misc{dcfe726914529b,
title = {Security Tests as Executable Specifications for LLM Code Generation: Benefits, Trade-offs, and Coverage Limits — arXiv:2608.09740},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.09740},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fe726914529b; sha256 45462a11c658056f53ecdac2e5ccd067e1769e5eec8e48bbfabd7bf4beb1b892},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.09740. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) can generate functionally useful code that remains vulnerable, while security-focused interventions may break intended behavior. We investigate security tests as executable specifications both before generation and during iterative repair. We develop SecTDD, a controlled test-feedback scaffold that separates three factors: whether tests are shown upfront, whether failed executions trigger revision, and how failures are selected and represented. The evaluation uses behavior-partitioned visible and hidden tests and byte-identical initial candidates for repair comparisons. Across 2,705 trajectories, 31 task instances, three secure-code benchmarks, 16 CWE categories, and two model families, showing all visible tests upfront increases hidden functional-and-security joint success by 19.3 percentage points on average, but improves only seven of nine benchmark-model conditions and harms two. In shared-candidate comparisons, structured feedback repairs 80 initially unsuccessful candidates with no joint regressions; fixed raw feedback repairs 83 but causes three regressions. Structured and raw feedback are otherwise nearly indistinguishable head-to-head (six wins, six losses, and 453 ties). Candidates that pass all visible tests still fail hidden behavior families under every common regime. These results show that executable feedback can repair secure-code generation, but its benefits depend on the model, task, feedback entry point, and especially test coverage. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-11 in cs.SE). Categorie arXiv: Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2608.09740 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T09:00:31.500Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-08-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).