Metraipolis › archive › dc_94aa285b3254
Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark — arXiv:2609.28449
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.28449
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T09:13:35.540Z
- SHA-256 of the archived text
8a3686a7621ca83ca4a15a81a483169acb69b148acca9759949a7c14ebea5467- Stable URL of this page
https://metraipolis.com/documento/dc_94aa285b3254
How to cite
Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark — arXiv:2609.28449 — arXiv (en), https://arxiv.org/abs/2609.28449 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_94aa285b3254 (Metraipolis archive dc_94aa285b3254, sha256 8a3686a7621ca83ca4a15a81a483169acb69b148acca9759949a7c14ebea5467)
BibTeX
@misc{dc94aa285b3254,
title = {Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark — arXiv:2609.28449},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.28449},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_94aa285b3254; sha256 8a3686a7621ca83ca4a15a81a483169acb69b148acca9759949a7c14ebea5467},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.28449. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) are increasingly used in coding tasks, but their ability to reason about code execution remains unclear. Existing repository-level QA benchmarks mainly evaluate static code understanding and often rely on LLM-based evaluation, while execution-reasoning benchmarks are mostly limited to snippets or functions. We introduce SWE-Flux, a repository-level benchmark for dynamic execution reasoning containing 480 execution-grounded instances across 12 real Python repositories, with gold answers automatically harvested from instrumented test executions rather than written manually or judged by LLMs. The benchmark covers singletest and multi-test questions over control flow, loops, program state, dataflow, exceptions, and program invariants. Evaluating five LLMs shows that this task remains challenging. The best model achieves only 37% accuracy. Models perform better on localized behavior such as invariants, intra-procedural control flow, exceptions, and simple loops, but struggle with dataflow, inter-procedural execution, precise state reasoning, and suite-level aggregation. Finally, we show that the oracle-harvesting pipeline can generate fresh benchmark variants using input perturbation. It successfully harvests valid variants for almost 90% of the selected instances, and the resulting variants are substantially more challenging for the evaluated models. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.AI). Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.28449 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T08:13:26.776Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).