Metraipolis › archive › dc_d30feaa94e3f

DatalogBench: Evaluating Large Language Models on Text-to-Datalog Synthesis — arXiv:2609.37233

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.37233
License (as archived)
CC0-1.0
Archived
2026-10-07T16:27:03.723Z
SHA-256 of the archived text
61b84a2fcc2bc6fc20c708748c2d590f675ab8f7ff6b84acc49cd9c95621b455
Stable URL of this page
https://metraipolis.com/documento/dc_d30feaa94e3f

How to cite

DatalogBench: Evaluating Large Language Models on Text-to-Datalog Synthesis — arXiv:2609.37233 — arXiv (en), https://arxiv.org/abs/2609.37233 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d30feaa94e3f (Metraipolis archive dc_d30feaa94e3f, sha256 61b84a2fcc2bc6fc20c708748c2d590f675ab8f7ff6b84acc49cd9c95621b455)
BibTeX
@misc{dcd30feaa94e3f,
  title = {DatalogBench: Evaluating Large Language Models on Text-to-Datalog Synthesis — arXiv:2609.37233},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.37233},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d30feaa94e3f; sha256 61b84a2fcc2bc6fc20c708748c2d590f675ab8f7ff6b84acc49cd9c95621b455},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37233. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Datalog underpins reasoning tasks such as program analysis, but its programs are hard to write. Existing synthesizers automate this task but require users to state their intent as input-output examples. Large language models (LLMs) suggest a more natural route, text-to-Datalog synthesis from a natural-language question, yet how well they do so has not been systematically evaluated. We present DatalogBench, a benchmark of 136 text-to-Datalog synthesis tasks curated from existing Datalog-based artifacts. Synthesized programs are graded by execution on held-out inputs against an oracle validated by mutation analysis. Across six LLMs and four prompting configurations, exact match peaks at 68.4%, and relation descriptions or an input-output example have only modest, model-dependent effects. Under direct prompting, most failures occur at compile time, typically because a model invents auxiliary predicates that it never declares or types consistently. Two coding agents reach up to 83.8% and eliminate nearly all such failures, leaving mostly semantic errors concentrated in recursive tasks. DatalogBench thus identifies recursive reasoning and decomposition as open challenges for current LLMs and agents, and offers a reliable, execution-grounded measure of both.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Programming Languages (cs.PL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.37233 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:53.961Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).