Metraipolis › archive › dc_79fb84d19647

MemDev-Bench: Does Memory Help Code Agents?Benchmarking Memory Systems over Real Repository Histories — arXiv:2610.27524

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.27524
License (as archived)
CC0-1.0
Archived
2026-10-21T07:06:37.205Z
SHA-256 of the archived text
077bce7c415335556a0e03f24c44980775f085621a2dfca6406a3c34f0403a22
Stable URL of this page
https://metraipolis.com/documento/dc_79fb84d19647

How to cite

MemDev-Bench: Does Memory Help Code Agents?Benchmarking Memory Systems over Real Repository Histories — arXiv:2610.27524 — arXiv (en), https://arxiv.org/abs/2610.27524 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_79fb84d19647 (Metraipolis archive dc_79fb84d19647, sha256 077bce7c415335556a0e03f24c44980775f085621a2dfca6406a3c34f0403a22)
BibTeX
@misc{dc79fb84d19647,
  title = {MemDev-Bench: Does Memory Help Code Agents?Benchmarking Memory Systems over Real Repository Histories — arXiv:2610.27524},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.27524},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_79fb84d19647; sha256 077bce7c415335556a0e03f24c44980775f085621a2dfca6406a3c34f0403a22},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.27524. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Code agents increasingly work on evolving real repositories, and memory systems aim to help them reuse earlier experience. However, such experience is diverse and evolving, so it remains unclear how memory systems organize it and thereby affect code agents. Existing benchmarks cannot answer this reliably because they lack either temporally valid histories or control over prior-task relevance. In this paper, we present MemDev-Bench to evaluate how memory systems affect code agents over diverse repository histories. It pairs each task with relevance-annotated earlier tasks from the same repository to build histories with controlled relevance. We also build an automated pipeline that reliably makes these earlier tasks executable and verifiable, producing 4,869 historical tasks for 555 base tasks. Based on MemDev-Bench, our systematic evaluation of memory systems across diverse development scenarios yields three main findings. (1) Memory mainly improves fine-grained localization and editing, increasing functional correctness. (2) Memory also causes failures on otherwise solvable tasks even with related history, and unrelated history tends to increase such failures. These failures mainly stem from agents reusing memory without reassessing its applicability. (3) Compared with using memory systems, directly providing agents with raw historical trajectories substantially reduces their steps. Further analysis suggests that agents save these steps by reusing process information that compressed memories tend to omit. These results suggest that future memory systems should preserve process information and reassess memory applicability against current task context.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.27524 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-21T07:05:43.418Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).