Metraipolis › archive › dc_01d4723082bb
Certified Selective Automation of LLM Agent Evaluation — arXiv:2609.34320
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.34320
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:53:23.116Z
- SHA-256 of the archived text
7e581fd19481cc29646e25e74c3bd53115c67da423a3f0ca0b570a907e08772e- Stable URL of this page
https://metraipolis.com/documento/dc_01d4723082bb
How to cite
Certified Selective Automation of LLM Agent Evaluation — arXiv:2609.34320 — arXiv (en), https://arxiv.org/abs/2609.34320 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_01d4723082bb (Metraipolis archive dc_01d4723082bb, sha256 7e581fd19481cc29646e25e74c3bd53115c67da423a3f0ca0b570a907e08772e)
BibTeX
@misc{dc01d4723082bb,
title = {Certified Selective Automation of LLM Agent Evaluation — arXiv:2609.34320},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.34320},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_01d4723082bb; sha256 7e581fd19481cc29646e25e74c3bd53115c67da423a3f0ca0b570a907e08772e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34320. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Evaluating LLM agents still ends with a human reading trajectories, because automatic judges carry no guarantee on how often they are wrong. We ask the operational question: what fraction of agent evaluation can a judge take over, with a certificate that the error rate among auto-decided trajectories stays below a budget alpha? Agent corpora resist the standard answer: many agents attempt the same tasks, so trajectories arrive in correlated clusters, and the i.i.d. certificates of existing selective-judging methods can overstate what is safe: a naive certificate can claim 98% automation while its realized error exceeds the budget in 17.5% of task resamples. We introduce a task-level bootstrap certificate that is valid in every regime we test while matching the naive certificate's coverage; finite-sample cluster-valid alternatives certify nothing at realistic task counts. Under this certificate, a 4B logprob judge trained with SFT and reject-weighted GRPO certifies 0.30-0.59 of evaluation on tool-use and web corpora at alpha=0.1, the only judge, among strongly elicited frontier models, certifying on both headline corpora. Certified coverage is predictable before training from base rate and discrimination alone (leave-one-corpus-out R^2=0.96). Finally, the certificate doubles as a self-training filter: pseudo-labels harvested inside certified regions have contamination bounded by alpha by construction (realized 0.000-0.041 across six harvests), letting a judge enter an unseen domain at in-domain strength with zero target training labels. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.34320 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T21:27:08.605Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).