Metraipolis › archive › dc_2c63773ea90f

What Does a Repair Agent Gain from Remembering Its Failure? An Exploratory Study on Real Python Bugs — arXiv:2610.41433

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.41433
License (as archived)
CC0-1.0
Archived
2026-11-03T08:13:54.161Z
SHA-256 of the archived text
8088368e3e8ee57e3c2267df38c6b1a881ff5bd8589323261b57616f3f789690
Stable URL of this page
https://metraipolis.com/documento/dc_2c63773ea90f

How to cite

What Does a Repair Agent Gain from Remembering Its Failure? An Exploratory Study on Real Python Bugs — arXiv:2610.41433 — arXiv (en), https://arxiv.org/abs/2610.41433 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2c63773ea90f (Metraipolis archive dc_2c63773ea90f, sha256 8088368e3e8ee57e3c2267df38c6b1a881ff5bd8589323261b57616f3f789690)
BibTeX
@misc{dc2c63773ea90f,
  title = {What Does a Repair Agent Gain from Remembering Its Failure? An Exploratory Study on Real Python Bugs — arXiv:2610.41433},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41433},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2c63773ea90f; sha256 8088368e3e8ee57e3c2267df38c6b1a881ff5bd8589323261b57616f3f789690},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41433. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

When a coding agent fails a repair task, it is often retried with a record of the failure: the transcript, a self-written lesson, or extracted test signals. We ask what such failure memory changes in the retry. On 75 BugsInPy bugs, we retried each of 181 failed attempts of gpt-5.4-mini under five conditions, with a predeclared analysis whose deviations we report. Two predeclared follow-ups repeated the plain-retry, transcript and lesson conditions with the larger gpt-5.4 and added a notice-only control. On the bug-revealing tests, every memory condition gained over a plain retry in point estimate (+5.5 to +9.9 points over 29.8%), but none significantly. Five further findings stand out. First, a model-written lesson showed no advantage over the transcript it summarises (-1.7 points, 95% CI [-8.7, +4.9]; -3.5 for gpt-5.4), although it costs an extra model call. Second, the transcript may do more than tell the agent that it failed: a notice alone produced no significant change, while the transcript resolved 12.2 points more failures than the notice for gpt-5.4-mini (CI [+4.2, +20.0], partial run of 156 failures; the only significant one of twelve predeclared contrasts, and not significant for gpt-5.4). Third, post hoc, fewer retries with memory ended without modifying any Python file, while the share of editing retries that touched a file the fix changes was near ceiling. Fourth, in a post hoc full-test-suite check, retries that had also seen the previous attempt's test results more often failed other tests that the reference fix passes, which shrinks their apparent gains. Fifth, outcomes varied far more between failures than between conditions, and we give the sample sizes a confirmatory study would need.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2610.41433 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T08:02:53.768Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).