Metraipolis › archive › dc_dc840bee0b9b

Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models — arXiv:2610.10478

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.10478
License (as archived)
CC0-1.0
Archived
2026-10-08T11:09:54.135Z
SHA-256 of the archived text
74c1fe2357c1e178ee43ae3035587596b195a45d43269b7a86f987ea357a7b2a
Stable URL of this page
https://metraipolis.com/documento/dc_dc840bee0b9b

How to cite

Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models — arXiv:2610.10478 — arXiv (en), https://arxiv.org/abs/2610.10478 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dc840bee0b9b (Metraipolis archive dc_dc840bee0b9b, sha256 74c1fe2357c1e178ee43ae3035587596b195a45d43269b7a86f987ea357a7b2a)
BibTeX
@misc{dcdc840bee0b9b,
  title = {Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models — arXiv:2610.10478},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.10478},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dc840bee0b9b; sha256 74c1fe2357c1e178ee43ae3035587596b195a45d43269b7a86f987ea357a7b2a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.10478. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

How can we predict which base checkpoint is worth an expensive round of agentic post-training? End-to-end pass@$K$ tests whether successful behavior already appears in a base model's distribution, but it is a poor fit for agentic coding: many base checkpoints cannot reliably produce the well-formed tool invocation required to complete a task end-to-end. Single-shot or short-horizon tasks avoid these tool-calling failures by collapsing a multi-step interaction into a fixed prompt and a single patch, but they sidestep the core capability we care about: maintaining coherent state over many tool-using steps as the repository evolves. To bridge this gap, we treat successful post-trained agent trajectories as a lookahead signal of base-model potential. Replaying each trajectory and rerunning tests after every code-changing step identifies the decisive step: the first step whose cumulative patch flips the repository from failing to passing, certifying that the recorded action solves the task given the prior context. Motivated by a coverage principle for agentic traces, we build three screens at this step that do not require a base checkpoint to drive the harness from a cold start: (i) Decisive-Action BPB (bits per byte) measures the probability mass on the certified action, (ii) Patch MCQ tests the checkpoint's choice between that action and alternatives rejected by the same verifier, and (iii) prefix-conditioned pass@$K$ evaluates support for functionally-correct generations and credits any continuation that the tests accept. Across ten pairs of public base and post-trained models, all three screens rank the cohort in close agreement with post-trained SWE-bench Verified pass@$1$. As our methods need only a benchmark's successful trajectories and its verifier, they can be applied to turn future agentic coding benchmarks into base-model evaluations.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2610.10478 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:14:05.990Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).