Metraipolis › archive › dc_3acc81ef40d7
MongoBench: Do LLMs Know What a MongoDB Query Will Do? — arXiv:2611.09065
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.09065
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-09T22:42:03.759Z
- SHA-256 of the archived text
9c67aa8d6e9d4065a62799b076f591c06cb5a84002c8fddf93f78ff157dc0f24- Stable URL of this page
https://metraipolis.com/documento/dc_3acc81ef40d7
How to cite
MongoBench: Do LLMs Know What a MongoDB Query Will Do? — arXiv:2611.09065 — arXiv (en), https://arxiv.org/abs/2611.09065 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3acc81ef40d7 (Metraipolis archive dc_3acc81ef40d7, sha256 9c67aa8d6e9d4065a62799b076f591c06cb5a84002c8fddf93f78ff157dc0f24)
BibTeX
@misc{dc3acc81ef40d7,
title = {MongoBench: Do LLMs Know What a MongoDB Query Will Do? — arXiv:2611.09065},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.09065},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3acc81ef40d7; sha256 9c67aa8d6e9d4065a62799b076f591c06cb5a84002c8fddf93f78ff157dc0f24},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.09065. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLM agents increasingly operate databases on their own: they generate queries and run them against live data. Two checks are commonly used to judge such an agent - whether it produces a valid query, and whether it reaches the right final answer - but neither tells us whether the agent knows what its query will actually do before it runs. We argue that this consequence prediction ability must be evaluated in its own right, and we build MongoBench to measure it: given a collection of documents and a MongoDB find() filter, a model must return the exact set of matching _ids, with ground truth from a real mongod server. The 300 items each compose three or four documented matching pitfalls and are proposed symmetrically by three vendors' models, so results stratify by item source. Evaluating four models - frontier systems from three vendors and one open-weight reasoner - accuracy ranges from 70.1% to 92.1%, a 22-point spread. Each proposer mispredicts 10-31% of the items it wrote itself, so producing a query does not imply predicting its effect. A failure-mode analysis shows errors are better predicted by which semantic pitfall a query exercises - range comparisons and null - than by surface measures such as document count or the number of composed rules. Prepending a verified rule sheet helps unevenly; executing the original filter against the engine yields reliable match sets, but delegation must be checked: on 28% of items (averaged over three runs) one model's executed filters never passed a canonical check against the intended query, and it answered about 22 of them wrong, while the other three passed on every item. We release the benchmark, the mongod oracle, and the evaluation code with all prompts at this https URL . Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE) ; Databases (cs.DB). Pagina del lavoro: https://arxiv.org/abs/2611.09065 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-09T22:34:02.189Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).