Metraipolis › archive › dc_a99700ebf996

Silent Instrument Errors Reorder Agent Leaderboards — arXiv:2611.04399

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04399
License (as archived)
CC0-1.0
Archived
2026-11-04T21:37:57.247Z
SHA-256 of the archived text
b560939eff33e601392b1c8c132fcd8689613e05ba226f3e2fc437432f942f69
Stable URL of this page
https://metraipolis.com/documento/dc_a99700ebf996

How to cite

Silent Instrument Errors Reorder Agent Leaderboards — arXiv:2611.04399 — arXiv (en), https://arxiv.org/abs/2611.04399 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a99700ebf996 (Metraipolis archive dc_a99700ebf996, sha256 b560939eff33e601392b1c8c132fcd8689613e05ba226f3e2fc437432f942f69)
BibTeX
@misc{dca99700ebf996,
  title = {Silent Instrument Errors Reorder Agent Leaderboards — arXiv:2611.04399},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04399},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a99700ebf996; sha256 b560939eff33e601392b1c8c132fcd8689613e05ba226f3e2fc437432f942f69},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04399. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On a public agent leaderboard, a general-purpose agent trails a scaffold built for an airline customer-service benchmark by 13 points on the tasks that touch its database, and a reader could conclude that generality costs accuracy. The gap comes from the grader. The general agent often ends its conversations in a way that never triggers grading, so only about half of its episodes were scored and the rest were stored as failures. Grading the ungraded half under the benchmark's own rule closes the pooled gap, without rerunning any agent. We audit two carefully maintained public leaderboards, HAL and tau2-bench, from their published artifacts alone and find three more instrument errors. A cost column counts nested model calls twice, unevenly across rows. Archived trajectories record grader-written actions as agent actions. Re-grading saved trajectories under a newer evaluator is itself a new measurement: a correct and announced fix reorders a leaderboard, and a mode check that misreads four files collapses their scores by 59 to 82 points while printing no error. Each produced a plausible number, and the HAL maintainers confirmed their three within five weeks. Together the cases show that the infrastructure of agent evaluation, meaning the code that grades, logs, replays and versions an evaluation, is part of the measurement and fails silently, so scores and orderings can move with no visible sign. These failure modes follow from how agent evaluation pipelines are built, and those pipelines change as fast as the agents they measure, so we expect more of them elsewhere. We propose cheap checks that expose the instrument: report the graded fraction, aggregate cost from root spans, tag the provenance of every recorded action, and pin evaluator versions with a published re-grade diff.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2611.04399 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T21:35:49.544Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).