Metraipolis › archive › dc_fcf0f89474ed
Eliciting Specifications: When Does Interactive Clarification Help? — arXiv:2610.22382
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.22382
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T08:42:45.466Z
- SHA-256 of the archived text
4a7826de83411d6a2b61afd71aabe27dcc4c7e78561ee400ee263ebbec7b9bda- Stable URL of this page
https://metraipolis.com/documento/dc_fcf0f89474ed
How to cite
Eliciting Specifications: When Does Interactive Clarification Help? — arXiv:2610.22382 — arXiv (en), https://arxiv.org/abs/2610.22382 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fcf0f89474ed (Metraipolis archive dc_fcf0f89474ed, sha256 4a7826de83411d6a2b61afd71aabe27dcc4c7e78561ee400ee263ebbec7b9bda)
BibTeX
@misc{dcfcf0f89474ed,
title = {Eliciting Specifications: When Does Interactive Clarification Help? — arXiv:2610.22382},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.22382},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fcf0f89474ed; sha256 4a7826de83411d6a2b61afd71aabe27dcc4c7e78561ee400ee263ebbec7b9bda},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22382. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLMs can generate Lean specifications from task descriptions, but those descriptions are often underspecified or ambiguous and may omit verifier-relevant edge cases, yielding imperfect formal specifications. We study whether an elicitation agent (EA) can improve verifier-checked Lean specifications by seeking clarification from a reference-grounded simulated developer before formalization. The agent decomposes the task, identifies unresolved decisions, asks bounded clarification questions, and uses the answers to write a refined natural-language specification that an LLM formalizer then translates into Lean. On 189 Verina Lean 4 specification-generation tasks, we compare single-shot generation with elicitation under question budgets of k=3 and k=10, allowing at most three or ten EA questions per task, across same-model and cross-model configurations. The same-model Claude Sonnet 4.6 pipeline has negative strict pass@1 estimates at both budgets, showing elicitation is not a uniform accuracy booster. In contrast, the Sonnet 4.6-to-GPT-5.6 and Sonnet 4.6-to-Gemini 3.8 Flash pipelines have positive strict estimates. The largest strict result reaches 65.6%, a +6.3-point change (+12 tasks), in the Sonnet 4.6-to-GPT-5.6 pipeline. Six of eight conditions have positive compile-conditional estimates, with a maximum cross-model difference of +4.9 points among compiling outputs. Treatment estimates also differ across the basic and advanced splits; the largest advanced-task change occurs in the Sonnet 4.6-to-GPT-5.6 configuration at k=10, from 42.0% to 50.6%. Failure analysis identifies postconditions as the largest source of degradation. These findings suggest future autoformalization systems should treat elicited natural-language specifications as intermediate artifacts and optimize jointly for semantic correctness among compiling outputs and reliable translation of clarified information into Lean. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.22382 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T08:39:10.100Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).