Metraipolis › archive › dc_01703f2bf08b
Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows — arXiv:2610.21109
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.21109
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T08:59:44.536Z
- SHA-256 of the archived text
26c6a6a6ac612ab7a978bfce34fc2a7a53de42c15ed2bcc45d71599006b849cf- Stable URL of this page
https://metraipolis.com/documento/dc_01703f2bf08b
How to cite
Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows — arXiv:2610.21109 — arXiv (en), https://arxiv.org/abs/2610.21109 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_01703f2bf08b (Metraipolis archive dc_01703f2bf08b, sha256 26c6a6a6ac612ab7a978bfce34fc2a7a53de42c15ed2bcc45d71599006b849cf)
BibTeX
@misc{dc01703f2bf08b,
title = {Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows — arXiv:2610.21109},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.21109},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_01703f2bf08b; sha256 26c6a6a6ac612ab7a978bfce34fc2a7a53de42c15ed2bcc45d71599006b849cf},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.21109. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
When humans collaborate with AI agents on long-horizon tasks, what happens to the agents' contributions? For solving long-horizon tasks, humans place greater importance on ongoing interaction with AI agents (e.g., revising plans) than on an agent's ability to complete the task in a single attempt. However, most agentic benchmarks have focused more on automation. Strong performance on such benchmarks does not imply effective real-world collaboration. Professional workflows offer a useful setting, as they often involve interdependent steps. To address this gap, we introduce Kolab, a new dataset of 115+ hours of screen recordings of 50 professionals from 40 occupations collaborating with agents on realistic, long-horizon problems. Kolab also contains participants' collaboration and output quality judgments. Our analysis reveals diverse collaboration patterns, highlighting the importance of when and how professionals coordinate with agents. A collaborative agent must do two things that automation benchmarks never test: anticipate whether experts will adopt, modify, or reject its contribution once it leaves the chat window to guide its next steps, and assess collaboration quality from the unfolding workflow without constant expert feedback. Our benchmark operationalizes both: (1) forecasting how professionals use agent contributions at diverse workflows, and (2) verifying their judgments of collaboration satisfaction and output quality. Findings show that frontier models, both agentic and non-agentic, overestimate collaboration with experts and struggle to distinguish between high- and low-quality collaboration. Longer workflow context does not consistently improve these predictions. As frontier agents cannot yet understand long-horizon collaboration with experts, we position Kolab as a valuable dataset for studying human-agent collaboration grounded in real professional workflows. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.21109 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T08:44:17.703Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).