Metraipolis › archive › dc_bee987ab6a32

VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks — arXiv:2609.23570

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.23570
License (as archived)
CC0-1.0
Archived
2026-10-09T06:50:37.123Z
SHA-256 of the archived text
1702024f35a46de49d15585697e7067d6f7c29b20a3d4f210235c96baa47468a
Stable URL of this page
https://metraipolis.com/documento/dc_bee987ab6a32

How to cite

VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks — arXiv:2609.23570 — arXiv (en), https://arxiv.org/abs/2609.23570 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bee987ab6a32 (Metraipolis archive dc_bee987ab6a32, sha256 1702024f35a46de49d15585697e7067d6f7c29b20a3d4f210235c96baa47468a)
BibTeX
@misc{dcbee987ab6a32,
  title = {VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks — arXiv:2609.23570},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.23570},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bee987ab6a32; sha256 1702024f35a46de49d15585697e7067d6f7c29b20a3d4f210235c96baa47468a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.23570. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Coding agents operate on real repository coding tasks, and persistent memory systems promise to reuse experience across tasks. Yet existing evaluations do not show whether those systems improve executable repository work. Repository benchmarks test code changes but do not isolate memory, while memory benchmarks score recall without measuring downstream coding outcomes. We introduce VibeMemBench, a benchmark for evaluating memory systems on 111 coding targets from 90 SWE-rebench V2 repositories and 3,634 history trajectories from the target repositories. The targets follow the SWE benchmark style and cover bug fixes, feature requests, interface changes, and configuration work. An agent edits each target codebase under a declared memory condition. Executable tests decide task resolution. Each target is retained only when injected history experience improves its executable outcome in a reference setting, so every target carries a prior experience whose usefulness is verified by execution in that setting. The frozen verified experience is then transferred to five held-out solvers. Direct injection raises observed task resolution on four of them by 1.1 to 4.5 percentage points while lowering agent steps on all five. Yet when four existing memory systems must construct and retrieve experience from the same history, eleven of twelve solver and system pairings fail to exceed the matched memory-off baseline. VibeMemBench exposes the gap between the useful experience that repository history holds and the experience existing memory systems deliver for repository coding tasks.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.CL).
Categorie arXiv: Software Engineering (cs.SE) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.23570 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:48:45.128Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).