Metraipolis › archive › dc_c3aa754ecb91

Testing-Driven Reliability Audit of Trajectory-Based Early Outcome Prediction for LLM Agents: Target-Specific Calibration Transfer Persists Within a Single Benchmark — arXiv:2609.25647

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.25647
License (as archived): CC0-1.0
Archived: 2026-10-08T13:45:11.451Z
SHA-256 of the archived text: faf770baf6dbc52fa966cd107527702b9c78a44597e672da143ac981ac4699c7
Stable URL of this page: https://metraipolis.com/documento/dc_c3aa754ecb91

How to cite

Testing-Driven Reliability Audit of Trajectory-Based Early Outcome Prediction for LLM Agents: Target-Specific Calibration Transfer Persists Within a Single Benchmark — arXiv:2609.25647 — arXiv (en), https://arxiv.org/abs/2609.25647 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c3aa754ecb91 (Metraipolis archive dc_c3aa754ecb91, sha256 faf770baf6dbc52fa966cd107527702b9c78a44597e672da143ac981ac4699c7)

BibTeX

@misc{dcc3aa754ecb91,
  title = {Testing-Driven Reliability Audit of Trajectory-Based Early Outcome Prediction for LLM Agents: Target-Specific Calibration Transfer Persists Within a Single Benchmark — arXiv:2609.25647},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.25647},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c3aa754ecb91; sha256 faf770baf6dbc52fa966cd107527702b9c78a44597e672da143ac981ac4699c7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.25647. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Predicting early outcomes based on trajectory can decrease the expenses associated with agent evaluation by terminating a run once the outcome becomes sufficiently predictable, assuming that the predictor's confidence is properly calibrated. Calibration is at risk when a predictor is applied to an agent on which it was never trained, but it is not known whether such transfer failures are broad across agent systems or concentrated in specific target agent/head combinations. Using public SWE-bench Verified trajectories and a frozen dual-head early-outcome prediction pipeline, we ran a leave-one-agent-out calibration audit, a shared-predictor leave-two-agents-out control, oracle prior correction, and a robustness battery over training cohorts, task resampling, task halves, jackknife, and thresholds. Fixed-scaffold TerminalBench analysis served as a pre-registered boundary test. Broad same-predictor pairwise heterogeneity was not supported; the median pairwise corrected-gap differences were 0.0180 (SUCCESS head, 45 pairs) and 0.0385 (FAILURE head, 35 pairs), and the pre-registered heterogeneity criterion was not met on either head. Two specific combinations, gpt-5-mini/SUCCESS and claude-opus-4.6/FAILURE, showed persistent calibration-transfer errors (median corrected gaps 0.1377 and 0.1107) without a sign reversal under any frozen control. TerminalBench did not establish cross-benchmark replication: the success target produced zero decisions (INDETERMINATE), and the failure target did not satisfy the pre-registered persistence criterion. Therefore, a strong target-specific calibration-transfer error can exist within one frozen environment, but the evidence does not establish that the error is intrinsic to the model or general across benchmarks.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-23 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.25647 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T13:18:38.104Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).