Metraipolis › archive › dc_381189388b07
SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics — arXiv:2609.11180
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.11180
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T05:23:44.902Z
- SHA-256 of the archived text
51c5b6c4d89968df9e0f53a343893e466c6e9a650a4836eea62ab7bce00d27ac- Stable URL of this page
https://metraipolis.com/documento/dc_381189388b07
How to cite
SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics — arXiv:2609.11180 — arXiv (en), https://arxiv.org/abs/2609.11180 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_381189388b07 (Metraipolis archive dc_381189388b07, sha256 51c5b6c4d89968df9e0f53a343893e466c6e9a650a4836eea62ab7bce00d27ac)
BibTeX
@misc{dc381189388b07,
title = {SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics — arXiv:2609.11180},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.11180},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_381189388b07; sha256 51c5b6c4d89968df9e0f53a343893e466c6e9a650a4836eea62ab7bce00d27ac},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.11180. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language model (LLM) coding agents constantly decide whether a version satisfies a constraint such as ^1.2.3 or >=2.0,<3, yet their grasp of version-constraint semantics has never been measured directly. We introduce SemVerBench, the first benchmark of LLM version-constraint resolution semantics across three ecosystems (npm, PEP 440, Cargo): 240 machine-checkable items with unique answers, built author-neutrally from four balanced sources (each ecosystem's official test suite plus three frontier LLM proposers) and labeled by a non-circular two-implementation oracle. Evaluating six frontier models, we find systematic, predictable per-mechanism blind spots: a partial-comparator carry rule (>1.2 means >=1.3.0) traps every model on Cargo (near 60%), and although standard PEP 440 prefix matching is universal, on zero-pad/post-release corner cases GPT-5.1 collapses (0/26) while Claude stays at 97-100% (verified on a 67-item oracle-validated set). Opus significantly outperforms all other models, and Sonnet outperforms the OpenAI models (McNemar). The failures look more like an activation/application gap than a knowledge gap: injecting the rule or a light correct hint recovers most errors, whereas interval decomposition does not, and models are at ceiling on the basic forms of the same rules. An author-stratified analysis finds no statistically significant self-favoritism. Because the task is verifiable and a free, 100%-correct resolver exists, tool delegation reaches ~100%: coding agents should delegate version resolution to a resolver rather than reason about versions in-head. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-11 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2609.11180 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T05:17:51.980Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).