Metraipolis › archive › dc_f5cec4fd0fc8
Does the Judge See What the Evaluation Requires? Input Assurance for LLM Agent Evaluation — arXiv:2611.02676
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02676
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:27:21.597Z
- SHA-256 of the archived text
effe8fd92f2601a2f702ca82c952ab12d839f0510a41b80240cc6320ca8a0daa- Stable URL of this page
https://metraipolis.com/documento/dc_f5cec4fd0fc8
How to cite
Does the Judge See What the Evaluation Requires? Input Assurance for LLM Agent Evaluation — arXiv:2611.02676 — arXiv (en), https://arxiv.org/abs/2611.02676 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f5cec4fd0fc8 (Metraipolis archive dc_f5cec4fd0fc8, sha256 effe8fd92f2601a2f702ca82c952ab12d839f0510a41b80240cc6320ca8a0daa)
BibTeX
@misc{dcf5cec4fd0fc8,
title = {Does the Judge See What the Evaluation Requires? Input Assurance for LLM Agent Evaluation — arXiv:2611.02676},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02676},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f5cec4fd0fc8; sha256 effe8fd92f2601a2f702ca82c952ab12d839f0510a41b80240cc6320ca8a0daa},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02676. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
An LLM judge assesses agent steps from text packets. A renderer converts recorded runs into these packets, while software checkers test records and packets against requirements. We ask what evidence shows a packet is what the evaluation requires and propose four input-assurance obligations: required content survives rendering, checkers cover their requirements, a failed check stops later use and the judge reads the bytes last checked. In one pipeline with 30 trajectories, matching hashes, passing checkers and successful staging (copying packets for the judge) coexisted with violations of all four obligations, the last two in integration tests. The renderer omitted every structured tool call, leaving 53 of 60 judged steps blank despite matching hashes. For one judge configuration, later context raised error flags by 13 more on error than on correct steps. Restoring every call reduced this contrast to 2, a reduction of 11 (0.37 per trajectory, 95% cluster-bootstrap interval 0.17 to 0.53). In packets ending at the judged step, that configuration abstained at least once on every blank step, while the other four sometimes abstained but also flagged blank error and correct steps. Both separately written checkers accepted an unmatched tool result. In integration tests, a stub replacing the judge read a failed build's packet and a copy changed after staging. Three added controls stopped 18 of 19 constructed fault cases before the stub read any packet. The remaining stale script never ran. We turn the obligations into four tests and plan to evaluate their transfer to independent pipelines. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2611.02676 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T11:02:52.199Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).