Metraipolis › archive › dc_718f0674f04a

Quizzing the Translation: A Prover-Grounded Evaluation Metric for NL$\rightarrow$FOL — arXiv:2609.33612

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.33612
License (as archived)
CC0-1.0
Archived
2026-10-07T21:01:35.001Z
SHA-256 of the archived text
82bb749856b59b43776821c19681f09edf234384d56f0f37d8d773c3c010e2d0
Stable URL of this page
https://metraipolis.com/documento/dc_718f0674f04a

How to cite

Quizzing the Translation: A Prover-Grounded Evaluation Metric for NL$\rightarrow$FOL — arXiv:2609.33612 — arXiv (en), https://arxiv.org/abs/2609.33612 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_718f0674f04a (Metraipolis archive dc_718f0674f04a, sha256 82bb749856b59b43776821c19681f09edf234384d56f0f37d8d773c3c010e2d0)
BibTeX
@misc{dc718f0674f04a,
  title = {Quizzing the Translation: A Prover-Grounded Evaluation Metric for NL$rightarrow$FOL — arXiv:2609.33612},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.33612},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_718f0674f04a; sha256 82bb749856b59b43776821c19681f09edf234384d56f0f37d8d773c3c010e2d0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33612. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A standard pipeline for symbolic reasoning over natural-language problems translates them into first-order logic and invokes a theorem prover. The translation step is the bottleneck: swap "every" for "some" and every inference that follows is corrupted. Yet today's metrics often score more broken translations higher than less broken ones, because BLEU, BERTScore, and Smatch++ reward surface overlap that the worst errors happen to preserve. We introduce SIV, which derives two kinds of probes from the target formula and uses a theorem prover to verify the candidate translation against each. Positive probes are statements the candidate must entail, which detect translations that drop content; contrastive probes are statements the candidate must not entail, which detect translations that assert more than the original. On a controlled pool of perturbed FOLIO translations, the severity of the error accounts for 80% of SIV's score variance, compared with at most 17% for any prior metric. Across six error classes on a disjoint pool, SIV scores the reference above the perturbed candidate in over 99% of pairs. Because each probe is labeled with what it tests, the failure pattern also supplies a labeled error trace, recovering the perturbation class at macro-F1 0.638, nearly double the score-only baseline. On 434 expert-audited real LLM translations, SIV attains the top AUC, uniquely detects and grades expert-labeled major errors, and abstains, rather than mis-scoring, on out-of-vocabulary translations.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Logic in Computer Science (cs.LO).
Pagina del lavoro: https://arxiv.org/abs/2609.33612 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:02.700Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).