Metraipolis › archive › dc_a5e70284bf8a

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization — arXiv:2608.06301

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.06301
License (as archived)
CC0-1.0
Archived
2026-10-25T21:31:27.649Z
SHA-256 of the archived text
ca420aea50cae49f4322b5645eea513f9181f15bf02672aeac7b448ff71a577c
Stable URL of this page
https://metraipolis.com/documento/dc_a5e70284bf8a

How to cite

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization — arXiv:2608.06301 — arXiv (en), https://arxiv.org/abs/2608.06301 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a5e70284bf8a (Metraipolis archive dc_a5e70284bf8a, sha256 ca420aea50cae49f4322b5645eea513f9181f15bf02672aeac7b448ff71a577c)
BibTeX
@misc{dca5e70284bf8a,
  title = {HarnessOpt-Bench: Evaluating LLMs at Harness Optimization — arXiv:2608.06301},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.06301},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a5e70284bf8a; sha256 ca420aea50cae49f4322b5645eea513f9181f15bf02672aeac7b448ff71a577c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.06301. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-07 in cs.CL).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.06301 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-25T21:24:38.280Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).