Metraipolis › archive › dc_042e5e765700

Beyond Type-checking: Towards Holistic Evaluation of Formal Specification Generation — arXiv:2610.10604

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.10604
License (as archived)
CC0-1.0
Archived
2026-10-09T05:53:33.833Z
SHA-256 of the archived text
b347d7258d161a87a5d4b23fd9e5daae8f6b4e227e521a340a102f4dee5667e7
Stable URL of this page
https://metraipolis.com/documento/dc_042e5e765700

How to cite

Beyond Type-checking: Towards Holistic Evaluation of Formal Specification Generation — arXiv:2610.10604 — arXiv (en), https://arxiv.org/abs/2610.10604 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_042e5e765700 (Metraipolis archive dc_042e5e765700, sha256 b347d7258d161a87a5d4b23fd9e5daae8f6b4e227e521a340a102f4dee5667e7)
BibTeX
@misc{dc042e5e765700,
  title = {Beyond Type-checking: Towards Holistic Evaluation of Formal Specification Generation — arXiv:2610.10604},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.10604},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_042e5e765700; sha256 b347d7258d161a87a5d4b23fd9e5daae8f6b4e227e521a340a102f4dee5667e7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.10604. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

When generating verifiable code, natural language requirements are mapped to machine checked code using LLMs and agentic workflows. A crucial component of this pipeline is specification generation (SpecGen), which produces a formal contract against which an agent can prove implementation correctness. Proof generation can obtain deterministic feedback from a theorem prover, but SpecGen lacks a definitive check that a generated specification captures the user's intent. A checked proof can therefore establish correctness against a specification that misrepresents the intended behaviour. We take a step towards holistic SpecGen evaluation with a unified dataset assembled from $350$ existing Lean tasks, including $189$ from VERINA and $161$ from CLEVER, and a framework covering formal validity, reference similarity and equivalence, and behavioural adequacy. We distinguish acceptance of required inputs from acceptance of valid outputs and rejection of invalid outputs, while making each metric's evidence scope explicit. Across four SpecGen configurations, restricting the generalized tree edit distance (GTED) comparison, a reference similarity measure, to $32$ jointly measurable VERINA tasks changes the VERINA configuration's position from second to fourth in mean similarity, showing the importance of measurement coverage. In an authored control, a specification achieves $100\%$ positive test recall and negative test rejection while accepting $0\%$ of required inputs. This demonstrates that perfect postcondition scores can miss an unusable input contract, motivating separate feedback on input coverage and output constraints.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.10604 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T05:47:01.204Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).