Metraipolis › archive › dc_ceb96e3ddbf8

ContractEval: Query-Conditioned Execution Matching for Procedural Instruction Conformance — arXiv:2609.09458

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.09458
License (as archived)
CC0-1.0
Archived
2026-10-16T07:24:24.696Z
SHA-256 of the archived text
698df4c99f175342c0a2deb0412c2b2df18a2b930b0089cbea0e891da8522774
Stable URL of this page
https://metraipolis.com/documento/dc_ceb96e3ddbf8

How to cite

ContractEval: Query-Conditioned Execution Matching for Procedural Instruction Conformance — arXiv:2609.09458 — arXiv (en), https://arxiv.org/abs/2609.09458 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ceb96e3ddbf8 (Metraipolis archive dc_ceb96e3ddbf8, sha256 698df4c99f175342c0a2deb0412c2b2df18a2b930b0089cbea0e891da8522774)
BibTeX
@misc{dcceb96e3ddbf8,
  title = {ContractEval: Query-Conditioned Execution Matching for Procedural Instruction Conformance — arXiv:2609.09458},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.09458},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ceb96e3ddbf8; sha256 698df4c99f175342c0a2deb0412c2b2df18a2b930b0089cbea0e891da8522774},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.09458. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As LLM agents move from answering questions to carrying out procedures, failures can be unwarranted rather than visibly wrong: the final response looks acceptable even though the system skipped the check, branch, dependency, or invariant that made the answer justified. Output-only evaluation sees the answer, and trace-aware judging sees activity, but neither identifies which obligations were active for the query. We introduce CONTRACTEVAL, a diagnostic framework for making those active obligations explicit. It represents procedural instructions as query-active obligations and matches them against response or trace evidence, turning omissions, wrong branches, ordering errors, extra actions, invariant breaches, and output-contract violations into distinct conformance failures. On a controlled suite of audited procedural contracts, output-only and trace-aware LLM judges miss many injected structural failures; under gold expected and observed graphs, ContractEval detects and localizes all of them. LLM-backed extraction preserves much of this signal but remains calibration-sensitive. ContractEval is therefore not a compliance guarantee; it makes procedural conformance auditable rather than implicit in final-answer quality.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-10 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.09458 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T07:00:16.188Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).