Metraipolis › archive › dc_a0a210ea9698

Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program — arXiv:2610.07003

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.07003
License (as archived)
CC0-1.0
Archived
2026-10-07T11:28:25.869Z
SHA-256 of the archived text
2470095a56d9d61e8c35591f5b86061db845724cca616b6a75c9079edb5d5a76
Stable URL of this page
https://metraipolis.com/documento/dc_a0a210ea9698

How to cite

Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program — arXiv:2610.07003 — arXiv (en), https://arxiv.org/abs/2610.07003 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a0a210ea9698 (Metraipolis archive dc_a0a210ea9698, sha256 2470095a56d9d61e8c35591f5b86061db845724cca616b6a75c9079edb5d5a76)
BibTeX
@misc{dca0a210ea9698,
  title = {Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program — arXiv:2610.07003},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.07003},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a0a210ea9698; sha256 2470095a56d9d61e8c35591f5b86061db845724cca616b6a75c9079edb5d5a76},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07003. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Firms increasingly evaluate deployed AI agents using repeated human ratings, yet observed score changes may reflect the measurement process as much as changes in the agent itself. Firms also rarely know how long an evaluation remains informative as deployed systems evolve. We study reliability as a latent state using 2,611 rubric-scored production interviews from a deployed voice-and-video interviewing agent, overlapping human reviewers, the agent's change log, and support tickets. Reviewer severity varied substantially: two reviewers assessing the same batches differed by 0.79 standard deviations on the composite index, and a shift in reviewer composition flattened the raw trend. After adjusting for reviewer effects and smoothing batch-level estimates, evaluated reliability increased by 0.53 standard deviations from March to August 2026. Reliability also shifted materially across logged deploys. At the estimated rate of unaccounted movement, forecast uncertainty reaches the magnitude of the largest observed deploy contrast after about five weeks, although this horizon is imprecisely estimated and based on a small number of step-like changes. An independent operational outcome moved in the same direction: interview-related support tickets fell by about 10% per month relative to technical-issue tickets. These data do not identify a causal effect of evaluation on performance. Instead, they show that longitudinal AI evaluation requires three checks: comparability across raters, evidence currency under drift, and corroboration with consequential operational outcomes.

Tema degli esploratori: economia dei servizi IA (elenco arXiv econ.GN, nuovi lavori del giorno).
Categorie arXiv: General Economics (econ.GN).
Pagina del lavoro: https://arxiv.org/abs/2610.07003 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:17:09.458Z dalla pagina https://arxiv.org/list/econ.GN/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).