Metraipolis › archive › dc_084b0541ffc2

Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It? — arXiv:2609.27891

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.27891
License (as archived)
CC0-1.0
Archived
2026-10-08T09:13:10.703Z
SHA-256 of the archived text
5445ba01471506f260b2f908be60f0eb16f91b19824a6a104ca0e85bb178bc0d
Stable URL of this page
https://metraipolis.com/documento/dc_084b0541ffc2

How to cite

Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It? — arXiv:2609.27891 — arXiv (en), https://arxiv.org/abs/2609.27891 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_084b0541ffc2 (Metraipolis archive dc_084b0541ffc2, sha256 5445ba01471506f260b2f908be60f0eb16f91b19824a6a104ca0e85bb178bc0d)
BibTeX
@misc{dc084b0541ffc2,
  title = {Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It? — arXiv:2609.27891},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.27891},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_084b0541ffc2; sha256 5445ba01471506f260b2f908be60f0eb16f91b19824a6a104ca0e85bb178bc0d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.27891. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Repository-level coding benchmarks have become the standard for evaluating coding agents, yet they inherently suffer from data leakage because they are built upon popular open-source repositories repeatedly used for training. Consequently, strong performance may reflect memorization of canonical repository cues rather than robust repository reasoning. We propose SchrodingerRepo (Schrödinger's Repository), an evaluation framework for testing coding agents under dynamically instantiated repository representations. Instead of repeatedly using a static representation of the test repository, SchrodingerRepo treats the test repository as an evaluation-time latent variable that is dynamically instantiated only when the agent enters the evaluation environment. The instantiated repository preserves the original executable behavior while eroding familiar cues such as naming conventions, file layouts, and implementation patterns through four transformation levels: problem statement reconstruction, namespace remapping, intra-file layout reordering, and functionality-preserving code rewriting. We evaluate popular LLMs on SWE-bench Verified and SWE-QA. Results show that removing familiar repository cues consistently degrades agent performance and substantially increases interaction costs across models. Further analysis reveals that the additional cost is primarily caused by increased difficulty in repository exploration and localization. These findings suggest that current coding agents may partially rely on memorized repository-side cues, highlighting the need for evaluation under dynamically instantiated repository representations.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.AI).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.27891 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T08:13:26.735Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).