Metraipolis › archive › dc_60dd57cd3118
EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction — arXiv:2609.02783
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.02783
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T04:16:57.077Z
- SHA-256 of the archived text
ed1549ac4f62b79ef739e4b5e2f60af9f98ffab1db2773237f06d4cd397f1769- Stable URL of this page
https://metraipolis.com/documento/dc_60dd57cd3118
How to cite
EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction — arXiv:2609.02783 — arXiv (en), https://arxiv.org/abs/2609.02783 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_60dd57cd3118 (Metraipolis archive dc_60dd57cd3118, sha256 ed1549ac4f62b79ef739e4b5e2f60af9f98ffab1db2773237f06d4cd397f1769)
BibTeX
@misc{dc60dd57cd3118,
title = {EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction — arXiv:2609.02783},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.02783},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_60dd57cd3118; sha256 ed1549ac4f62b79ef739e4b5e2f60af9f98ffab1db2773237f06d4cd397f1769},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.02783. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Evaluating LLM agents is essential for guiding their development, yet it has grown prohibitively expensive: a single pass of a frontier model over an agentic benchmark can cost hundreds to thousands of dollars, a price paid repeatedly across iterative development cycles. Prior efforts, centered on benchmark distillation, reduce the number of evaluation tasks but leave the cost of executing each retained task untouched. In this work, we introduce early outcome prediction, a complementary axis of efficiency that instead cuts cost within each task. Our key insight is that an agent's final outcome is often evident from its intermediate behavior well before execution completes. We instantiate this idea in EarlyEval, a lightweight framework that trains a pair of LightGBM success and failure classifiers over behavioral, textual, and reference-solution features, and halts an agent run the moment either classifier crosses a calibrated confidence threshold, adding negligible per-step overhead. Across three benchmarks, SWE-bench Verified, TerminalBench, and Toolathlon, EarlyEval can eliminate 13%-26% of agent steps and up to 44.1% input tokens and 29.4% output tokens at 89%-97% prediction accuracy, while perturbing per-agent resolve rates by only one to two percentage points on average. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-03 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.02783 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T03:26:49.814Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-03?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).