Metraipolis › archive › dc_e9c34871b127

Verified but Vacuous: Three Failure Surfaces of LLM-Driven Verified Code Generation — arXiv:2611.05590

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.05590
License (as archived)
CC0-1.0
Archived
2026-11-05T18:16:39.160Z
SHA-256 of the archived text
7f7499c79036dcb9ae9b139109d44892d4dac91551504a184d1d789f25a84cee
Stable URL of this page
https://metraipolis.com/documento/dc_e9c34871b127

How to cite

Verified but Vacuous: Three Failure Surfaces of LLM-Driven Verified Code Generation — arXiv:2611.05590 — arXiv (en), https://arxiv.org/abs/2611.05590 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e9c34871b127 (Metraipolis archive dc_e9c34871b127, sha256 7f7499c79036dcb9ae9b139109d44892d4dac91551504a184d1d789f25a84cee)
BibTeX
@misc{dce9c34871b127,
  title = {Verified but Vacuous: Three Failure Surfaces of LLM-Driven Verified Code Generation — arXiv:2611.05590},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.05590},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e9c34871b127; sha256 7f7499c79036dcb9ae9b139109d44892d4dac91551504a184d1d789f25a84cee},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05590. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

What should ``verified'' mean when evaluating LLM-generated code? Many existing pipelines report a single verification number. We argue this collapses three distinct properties measured in a conditional audit sequence: verifier acceptance (Surface A), \emph{syntactic specification binding (Surface B: does the proof obligation bind to the spec?), and \emph{semantic specification fidelity} (Surface C: does the spec capture the informal requirement?). Two empirical demonstrations show the surfaces separate. \textbf{First ($A \neq B$), via Li@Opus (a same-model reproduction of \citet{li2025dafny}'s Dafny-as-intermediate-representation (IR) pipeline under Claude Opus 4.7) on three Python benchmarks}: it reaches \textbf{99.1\% verifier acceptance} on HumanEval while only \textbf{1.9\%} carry a certificate binding to a spec-defined predicate or function. \textbf{Second ($B \neq C$), via VCG on two Rust benchmarks}: we use VCG (which is our own in-house Dafny-as-intermediate pipeline) on Rust because the Rust target lets us enforce fully-verifiable output (every component ships as a Dafny certificate, ensures-binding gate passing by construction). Even inside that construction, \textbf{11.48\% (MBPP-Rust) and 14.82\% (VeriContest LC slice) of structurally-verified samples still fail hidden tests}. On the LC slice of VeriContest, \textbf{VCG's reported End2End rate is 7.79\% (535/6870).} The cited external rates are 5.51\% (GPT-5.5) and 1.88\% (single-shot Opus 4.7). These give descriptive differences of 2.3 percentage points and 4.14$\times$, respectively; task-budget, repair and harness differences prevent a matched-performance claim.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.05590 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:12:48.990Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).