Metraipolis › archive › dc_7647af8da03a
ProofJudge: Tool-Grounded LLM Evaluation of Formal Proof Quality in Mathlib — arXiv:2608.20432
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.20432
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T10:18:36.879Z
- SHA-256 of the archived text
3934b7a57b9a4f13dcf96d0d337b2485693481474370367a1d29d6378792d4fd- Stable URL of this page
https://metraipolis.com/documento/dc_7647af8da03a
How to cite
ProofJudge: Tool-Grounded LLM Evaluation of Formal Proof Quality in Mathlib — arXiv:2608.20432 — arXiv (en), https://arxiv.org/abs/2608.20432 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7647af8da03a (Metraipolis archive dc_7647af8da03a, sha256 3934b7a57b9a4f13dcf96d0d337b2485693481474370367a1d29d6378792d4fd)
BibTeX
@misc{dc7647af8da03a,
title = {ProofJudge: Tool-Grounded LLM Evaluation of Formal Proof Quality in Mathlib — arXiv:2608.20432},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.20432},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7647af8da03a; sha256 3934b7a57b9a4f13dcf96d0d337b2485693481474370367a1d29d6378792d4fd},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.20432. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Formal proofs in Lean 4 that pass the kernel's type checker can nonetheless vary widely in quality. We introduce ProofJudge, an agentic LLM-as-judge system that scores formal proof quality along five dimensions beyond correctness: library leverage, automation fit, structural clarity, statement quality, and Mathlib conventions. We evaluate ProofJudge on a novel dataset of 218 declarations drawn from distinct Mathlib PRs. The judge agent is grounded by tool access to the commit the PR is applied to, enabling it to query the library state when scoring. A judge is considered aligned with human preferences when it rates the version of the PR Mathlib accepted above the initial version that was sent back for revision. All six judge models evaluated recover the reviewers' preference well above chance, from 80.8% to 63.5%, and two open-weight judges reach roughly 70% at a tenth of the best judge's cost. We release the judge harness, evaluation dataset, and evaluation traces as open-source artifacts to support further research. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-24 in cs.CL). Categorie arXiv: Logic in Computer Science (cs.LO) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.20432 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T10:17:37.608Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).