Metraipolis › archive › dc_f8e69ea6e04a
What Does a Repair Agent Gain from Remembering Its Failure? An Exploratory Study on Real Python Bugs — arXiv:2610.41433
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.41433
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-02T04:54:22.137Z
- SHA-256 of the archived text
d144e06d6b0aafa2d980ed29c9bf2999edcba5e72a941297336308717d3baf6b- Stable URL of this page
https://metraipolis.com/documento/dc_f8e69ea6e04a
How to cite
What Does a Repair Agent Gain from Remembering Its Failure? An Exploratory Study on Real Python Bugs — arXiv:2610.41433 — arXiv (en), https://arxiv.org/abs/2610.41433 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f8e69ea6e04a (Metraipolis archive dc_f8e69ea6e04a, sha256 d144e06d6b0aafa2d980ed29c9bf2999edcba5e72a941297336308717d3baf6b)
BibTeX
@misc{dcf8e69ea6e04a,
title = {What Does a Repair Agent Gain from Remembering Its Failure? An Exploratory Study on Real Python Bugs — arXiv:2610.41433},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.41433},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f8e69ea6e04a; sha256 d144e06d6b0aafa2d980ed29c9bf2999edcba5e72a941297336308717d3baf6b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41433. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
When a coding agent fails a repair task, it is often retried with a record of the failure: the transcript, a self-written lesson, or extracted test signals. We ask what such failure memory changes in the retry. On 75 BugsInPy bugs, we retried each of 181 failed attempts of gpt-5.4-mini under five conditions, with a predeclared analysis whose deviations we report. Two predeclared follow-ups repeated the plain-retry, transcript and lesson conditions with the larger gpt-5.4 and added a notice-only control. On the bug-revealing tests, every memory condition gained over a plain retry in point estimate (+5.5 to +9.9 points over 29.8%), but none significantly. Five further findings stand out. First, a model-written lesson showed no advantage over the transcript it summarises (-1.7 points, 95% CI [-8.7, +4.9]; -3.5 for gpt-5.4), although it costs an extra model call. Second, the transcript may do more than tell the agent that it failed: a notice alone produced no significant change, while the transcript resolved 12.2 points more failures than the notice for gpt-5.4-mini (CI [+4.2, +20.0], partial run of 156 failures; the only significant one of twelve predeclared contrasts, and not significant for gpt-5.4). Third, post hoc, fewer retries with memory ended without modifying any Python file, while the share of editing retries that touched a file the fix changes was near ceiling. Fourth, in a post hoc full-test-suite check, retries that had also seen the previous attempt's test results more often failed other tests that the reference fix passes, which shrinks their apparent gains. Fifth, outcomes varied far more between failures than between conditions, and we give the sample sizes a confirmatory study would need. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2610.41433 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-02T04:45:27.639Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).