Metraipolis › archive › dc_d4f4fa0f1df9
Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations — arXiv:2608.11323
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.11323
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T05:15:18.719Z
- SHA-256 of the archived text
87e4618c6f406095b62de0702ed390e390bd8ef632a3e9291ef48fb1c97e7cdf- Stable URL of this page
https://metraipolis.com/documento/dc_d4f4fa0f1df9
How to cite
Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations — arXiv:2608.11323 — arXiv (en), https://arxiv.org/abs/2608.11323 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d4f4fa0f1df9 (Metraipolis archive dc_d4f4fa0f1df9, sha256 87e4618c6f406095b62de0702ed390e390bd8ef632a3e9291ef48fb1c97e7cdf)
BibTeX
@misc{dcd4f4fa0f1df9,
title = {Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations — arXiv:2608.11323},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.11323},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d4f4fa0f1df9; sha256 87e4618c6f406095b62de0702ed390e390bd8ef632a3e9291ef48fb1c97e7cdf},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.11323. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Enterprise practitioners read agent leaderboards as if they ranked agent capability. We show, across three open agent-trace benchmarks (TheAgentCompany, $\tau^2$-bench, and AppWorld), that the agent main effect accounts for less than 3% of total variance in every dataset and check type, while the agent-by-task interaction accounts for 7-23%. Leaderboards rank specialization, not capability. We arrive at this through a four-facet Generalizability Theory variance decomposition, fit with three estimators (Henderson Method-I, REML via lme4, and a Bayesian binomial GLMM) that agree to three decimal places. Four further findings sharpen what the leaderboard is hiding. First, aggregate reliability collapses on the hardest task quartile: $E\rho^2$ on $\tau^2$ action_checks falls from 0.752 to 0.000. Second, training-cell reliability negatively correlates with held-out reliability ($r = -0.90$ on $\tau^2$), meaning the designs that look most reliable replicate worst. Third, population-level diagnostics transfer across enterprise benchmarks (capability-gap ratio stable at 0.35-0.40) but per-family agent rankings invert. Fourth, on the MAST failure taxonomy, trace-level mode profiles are idiosyncratic (MAE = 0.261) while cell-level profiles generalise (MAE = 0.056, $r = 0.83$). We package these into Deployment Decision Reliability (DDR), a one-page reporting discipline that turns the variance-component table into five decisions an enterprise buyer can defend. All code, data loaders, and fit artifacts are released under an open-source license. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-13 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.11323 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T05:08:46.618Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).