Metraipolis › archive › dc_80c7dc8a9d94

RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents — arXiv:2609.04898

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.04898
License (as archived)
CC0-1.0
Archived
2026-10-16T22:26:47.948Z
SHA-256 of the archived text
becd3843c63c25ce1a3a1bd51d736ea50fdfd82dd02867293c61872ff44ac583
Stable URL of this page
https://metraipolis.com/documento/dc_80c7dc8a9d94

How to cite

RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents — arXiv:2609.04898 — arXiv (en), https://arxiv.org/abs/2609.04898 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_80c7dc8a9d94 (Metraipolis archive dc_80c7dc8a9d94, sha256 becd3843c63c25ce1a3a1bd51d736ea50fdfd82dd02867293c61872ff44ac583)
BibTeX
@misc{dc80c7dc8a9d94,
  title = {RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents — arXiv:2609.04898},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.04898},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_80c7dc8a9d94; sha256 becd3843c63c25ce1a3a1bd51d736ea50fdfd82dd02867293c61872ff44ac583},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04898. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Repository-scale refactoring requires coding agents to propagate a single change across many interdependent files without altering program behavior, yet to our knowledge no existing harness isolates the design choices that determine agent success on this task. We present RefactorPlatform, an open-source evaluation harness that holds the environment fixed and varies each design axis explicitly: model backbone (via OpenRouter and GitHub Copilot CLI), execution regime (baseline, retrieval-augmented, and multi-agent), and prompt specificity. Each run executes in an isolated workspace with live terminal streaming, per-task logging of tokens, diffs, and transcripts, AST-based verification, and exportable telemetry for audit and reproduction. Demonstrating the platform on 100 multi-file RefactorBench tasks across four model families, we illustrate the analyses it supports: AST-aware chunking outperforms naive token-window chunking by 25-30 percentage points (pp) across prompt modes, whereas naive retrieval falls below the retrieval-free baseline; a lean retrieval-augmented single agent (86%) beats the sub-agent configuration we evaluated (66%) on matched tasks with no task passing under delegation that fails under retrieval; and retrieval's accuracy gains absorb its token overhead, leaving cost per successful refactoring unchanged. RefactorPlatform is open-sourced to make refactoring-agent evaluation reproducible and auditable.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-07 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.04898 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T22:18:11.547Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).