Metraipolis › archive › dc_cbba8f4289ef
SCRBench: Beyond Equivalence Checking for Semantic Change Reasoning in Code — arXiv:2610.41562
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.41562
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-02T07:34:53.396Z
- SHA-256 of the archived text
0a694e64612715975d9ce5d78f550db69d512bffb7934b156c8a625c19058601- Stable URL of this page
https://metraipolis.com/documento/dc_cbba8f4289ef
How to cite
SCRBench: Beyond Equivalence Checking for Semantic Change Reasoning in Code — arXiv:2610.41562 — arXiv (en), https://arxiv.org/abs/2610.41562 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cbba8f4289ef (Metraipolis archive dc_cbba8f4289ef, sha256 0a694e64612715975d9ce5d78f550db69d512bffb7934b156c8a625c19058601)
BibTeX
@misc{dccbba8f4289ef,
title = {SCRBench: Beyond Equivalence Checking for Semantic Change Reasoning in Code — arXiv:2610.41562},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.41562},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cbba8f4289ef; sha256 0a694e64612715975d9ce5d78f550db69d512bffb7934b156c8a625c19058601},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41562. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Understanding how code changes affect program semantics is a core aspect of code reasoning in language models. It is commonly evaluated by asking whether two programs are semantically equivalent, but a yes/no answer does not reveal whether a model knows which code change caused a semantic difference, or which sequence of code changes produced the modified program. We study two tasks that ask these questions directly, diagnosing semantic change and reconstructing sequences of code changes, alongside detecting semantic equivalence. We build SCRBench, 67,658 Python program pairs generated by applying sequences of code changes that preserve or alter program semantics, and evaluate five language models on a stratified subset of 4,500 pairs. Detecting semantic equivalence exposes weaker models, whose accuracy declines as changes accumulate, but the strongest models remain near the ceiling. Diagnosing semantic change reveals weaknesses that equivalence accuracy hides: a model with 94.71% equivalence accuracy diagnoses only 54.44% of semantic changes. Reconstructing sequences of code changes is much harder: the best model reconstructs the exact sequence for only 39.27% of pairs, and no model exceeds 1.11% when five changes are applied. High equivalence accuracy therefore overestimates how well models reason about the semantic effects of code changes. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.41562 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-02T07:27:12.665Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).