Metraipolis › archive › dc_e41dabfd2813
Harder Than the Task: From Technical to Organizational Difficulty in AI Benchmarking — arXiv:2611.03494
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.03494
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:08:46.512Z
- SHA-256 of the archived text
4843fd6b2914fb708236d575aa973258a2b379b1da9f1e5b549c3788e88cfaea- Stable URL of this page
https://metraipolis.com/documento/dc_e41dabfd2813
How to cite
Harder Than the Task: From Technical to Organizational Difficulty in AI Benchmarking — arXiv:2611.03494 — arXiv (en), https://arxiv.org/abs/2611.03494 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e41dabfd2813 (Metraipolis archive dc_e41dabfd2813, sha256 4843fd6b2914fb708236d575aa973258a2b379b1da9f1e5b549c3788e88cfaea)
BibTeX
@misc{dce41dabfd2813,
title = {Harder Than the Task: From Technical to Organizational Difficulty in AI Benchmarking — arXiv:2611.03494},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.03494},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e41dabfd2813; sha256 4843fd6b2914fb708236d575aa973258a2b379b1da9f1e5b549c3788e88cfaea},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03494. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
AI benchmarks have made remarkable progress in measuring the ability of AI systems to perform increasingly demanding cognitive and technical tasks. Yet, many of these benchmarks treat technical tasks as the unit of analysis, while much economically valuable work takes place within organizations, where tasks are embedded in a system of relations. Within organizations, successful task performance depends not only on solving the technical demands of the task but also on navigating the interdependence of activities, distributed knowledge, differentiated authority, temporal constraints, conflicting interests, and changing environmental conditions while the task is underway. Accordingly, we argue that organizational task difficulty constitutes a distinct and insufficiently recognized dimension of AI capability, and that existing approaches to benchmarking provide an incomplete basis for assessing how effectively AI agents can perform real-world organizational work. By drawing on organization theory, we develop a taxonomy of the structure of work that characterizes the conditions that agents face in organizational work. Applying this taxonomy to analyze existing benchmarks, we find limited and uneven coverage of organizational dimensions, with fewer than one quarter of prominent benchmarks claiming any of them. We therefore argue that AI benchmarks should increasingly move beyond the evaluation of technically difficult tasks toward the systematic assessment of agents' ability to navigate the organizational conditions under which economically valuable work takes place. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Computers and Society (cs.CY) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.03494 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T10:59:46.169Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).