Metraipolis › archive › dc_ea16f315d8a9
Beyond "AI Helps Humans": Decision-Targeted Evaluation Design for Human-Agent Teams in the Agentic Era — arXiv:2609.05527
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.05527
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T13:46:51.085Z
- SHA-256 of the archived text
89d4050deb99b8435a148e63b56d9a339d822ef4cbeccb1a3e723d976791c1e4- Stable URL of this page
https://metraipolis.com/documento/dc_ea16f315d8a9
How to cite
Beyond "AI Helps Humans": Decision-Targeted Evaluation Design for Human-Agent Teams in the Agentic Era — arXiv:2609.05527 — arXiv (en), https://arxiv.org/abs/2609.05527 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ea16f315d8a9 (Metraipolis archive dc_ea16f315d8a9, sha256 89d4050deb99b8435a148e63b56d9a339d822ef4cbeccb1a3e723d976791c1e4)
BibTeX
@misc{dcea16f315d8a9,
title = {Beyond "AI Helps Humans": Decision-Targeted Evaluation Design for Human-Agent Teams in the Agentic Era — arXiv:2609.05527},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.05527},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ea16f315d8a9; sha256 89d4050deb99b8435a148e63b56d9a339d822ef4cbeccb1a3e723d976791c1e4},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.05527. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Wherever a coding agent works under engineer supervision, or a clinical model assists a radiologist, the deployment question is whether to keep the human-AI workflow or replace it with the human alone or the agent alone. The human-AI workflow is worth keeping only if it beats both of those alternatives. Yet once it is deployed, neither alternative outcome is observed: recovering one means replaying the task under that alternative, and every replay costs expert time or compute. Under a fixed replay budget, the design question is therefore which tasks should be more likely to receive a human-only replay, and which an agent-only replay. Existing methods do not directly target this decision. Agent benchmarks do not choose which missing baseline to measure, variance-based sampling ignores which of the two comparisons is closer to failing, and Bayesian information methods focus on learning model parameters instead of making the deployment decision. We propose TEAM-Design, a rule that gives every task two replay probabilities, one per baseline. It raises a probability where the missing baseline outcome is hard to predict from what is already known about the task and where that comparison is harder to establish, and lowers it where replay is expensive. We prove that the rule solves this budgeted design problem, and that drawing the replays at random from recorded probabilities still controls the chance of wrongly declaring that the workflow beats both. We reanalyze 6 clinical settings, where no human-AI workflow beats both alternatives, and a coding benchmark, where one does, then evaluate TEAM-Design on synthetic designs and on a semi-synthetic design built from a real chest X-ray reader study. TEAM-Design works best when one of the two comparisons is clearly harder to settle than the other, and can do worse than variance-based allocation when the two are similarly difficult. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.HC). Categorie arXiv: Artificial Intelligence (cs.AI) ; Human-Computer Interaction (cs.HC). Pagina del lavoro: https://arxiv.org/abs/2609.05527 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T13:28:43.158Z dalla pagina https://arxiv.org/catchup/cs.HC/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).