Metraipolis › archive › dc_df8b15015e10

Evo-Bench: Can Language Models Improve Agent Harness? — arXiv:2608.09096

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.09096
License (as archived)
CC0-1.0
Archived
2026-10-19T08:50:15.583Z
SHA-256 of the archived text
dc14870814902842875ea02c73398471b7115e62ebc4fa54026724baf009929d
Stable URL of this page
https://metraipolis.com/documento/dc_df8b15015e10

How to cite

Evo-Bench: Can Language Models Improve Agent Harness? — arXiv:2608.09096 — arXiv (en), https://arxiv.org/abs/2608.09096 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_df8b15015e10 (Metraipolis archive dc_df8b15015e10, sha256 dc14870814902842875ea02c73398471b7115e62ebc4fa54026724baf009929d)
BibTeX
@misc{dcdf8b15015e10,
  title = {Evo-Bench: Can Language Models Improve Agent Harness? — arXiv:2608.09096},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.09096},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_df8b15015e10; sha256 dc14870814902842875ea02c73398471b7115e62ebc4fa54026724baf009929d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.09096. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large Language Models (LLMs) have driven rapid progress in autonomous agents, yet standard evaluations remain confined to static task solving. An emerging frontier is harness evolution---the agent's capacity to autonomously optimize its own operating harness. However, systematically benchmarking this capability remains challenging, as existing evaluations fail to isolate harness improvements from base model strength, prevent task-specific overfitting, or capture long-horizon iterative research. To address these challenges, we introduce Evo-Bench, the first benchmark designed to evaluate models' intrinsic harness-evolving capabilities across Search, Office, and General agent domains. To rigorously isolate this capability, Evo-Bench employs a novel harness-guided construction framework: it leverages auxiliary-task evolution to identify tasks genuinely sensitive to framework improvements, followed by sensitivity-aware stratified splitting to ensure robust cross-suite generalization. Extensive evaluations across nine frontier and open-weight models reveal that top models achieve massive absolute gains reaching 16.6 points, closely approaching state-of-the-art human-engineered baselines. Crucially, while autonomous evolution outpeforms artificial harness in General tasks and excels in Search tasks, it struggles in Office tasks that demand highly specific processing workflows. Furthermore, our analysis exposes critical temporal anomalies like early saturation, while demonstrating that the synthesized harnesses act as highly transferable reasoning structures, consistently boosting diverse policy models.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-11 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.09096 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:49:41.912Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).