Metraipolis › archive › dc_57a3ea994ead

Correct Verdicts, Flawed Reasoning: Structured Auditing of LLM-based Vulnerability Reasoning — arXiv:2610.06366

Source
arXiv
Original
https://arxiv.org/abs/2610.06366
License (as archived)
CC0-1.0
Archived
2026-10-06T05:29:58.716Z
SHA-256 of the archived text
300bdf9c1bb88d9bffe7c90e690447e518350441a7ce880ffccc69130bac339f
Stable URL of this page
https://metraipolis.com/documento/dc_57a3ea994ead

How to cite

Correct Verdicts, Flawed Reasoning: Structured Auditing of LLM-based Vulnerability Reasoning — arXiv:2610.06366 — arXiv, https://arxiv.org/abs/2610.06366 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_57a3ea994ead (Metraipolis archive dc_57a3ea994ead, sha256 300bdf9c1bb88d9bffe7c90e690447e518350441a7ce880ffccc69130bac339f)
BibTeX
@misc{dc57a3ea994ead,
  title = {Correct Verdicts, Flawed Reasoning: Structured Auditing of LLM-based Vulnerability Reasoning — arXiv:2610.06366},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.06366},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_57a3ea994ead; sha256 300bdf9c1bb88d9bffe7c90e690447e518350441a7ce880ffccc69130bac339f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.06366. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large Language Models (LLMs) are increasingly deployed for automated software vulnerability analysis. Binary classification alone is insufficient; practitioners need explanations to triage bugs and engineer patches. Standard practice relies on Chain-of-Thought (CoT) prompting, but free-form reasoning allows models to obscure logical leaps, hallucinated execution steps, and internal inconsistencies behind plausible prose. Our manual audit reveals that approximately 60% of correct vulnerability verdicts are accompanied by fabricated or unverifiable claims, and free-form explanations allow reasoning errors to evade LLM-as-a-judge evaluation. We present Vulnerability Explanation Reasoning Auditor (VERA), an automated framework for auditing LLM vulnerability reasoning. Rather than accepting free-form text, VERA asks models to output a Structured Reasoning Record (SRR) encoding tracked pointers, memory operations, and state transitions in machine-readable fields. A multi-stage judge audits each SRR against eight reasoning failure modes using deterministic checks, with LLM calls reserved for semantic interpretation. The standardized SRR schema also enables automated mutation testing to benchmark judges at scale without human annotation. Our evaluation shows reasoning flaws occur in correct verdicts just as frequently as incorrect ones, and VERA exposes 87% of reasoning errors that free-form LLM-as-judge systematically miss.

Tema degli esploratori: fiducia digitale e crittografia (elenco arXiv cs.CR, nuovi lavori del giorno).
Categorie arXiv: Cryptography and Security (cs.CR) ; Machine Learning (cs.LG); Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2610.06366 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:26:35.375Z dalla pagina https://arxiv.org/list/cs.CR/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).