Metraipolis › archive › dc_463bac304d19
Harder Than the Task: From Technical to Organizational Difficulty in AI Benchmarking — arXiv:2611.03494
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.03494
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:20:27.938Z
- SHA-256 of the archived text
0ee59c25c8c975a4d6337c824b608b5273344fee9fe7083d7462385380f40de2- Stable URL of this page
https://metraipolis.com/documento/dc_463bac304d19
How to cite
Harder Than the Task: From Technical to Organizational Difficulty in AI Benchmarking — arXiv:2611.03494 — arXiv (en), https://arxiv.org/abs/2611.03494 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_463bac304d19 (Metraipolis archive dc_463bac304d19, sha256 0ee59c25c8c975a4d6337c824b608b5273344fee9fe7083d7462385380f40de2)
BibTeX
@misc{dc463bac304d19,
title = {Harder Than the Task: From Technical to Organizational Difficulty in AI Benchmarking — arXiv:2611.03494},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.03494},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_463bac304d19; sha256 0ee59c25c8c975a4d6337c824b608b5273344fee9fe7083d7462385380f40de2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03494. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
AI benchmarks have made remarkable progress in measuring the ability of AI systems to perform increasingly demanding cognitive and technical tasks. Yet, many of these benchmarks treat technical tasks as the unit of analysis, while much economically valuable work takes place within organizations, where tasks are embedded in a system of relations. Within organizations, successful task performance depends not only on solving the technical demands of the task but also on navigating the interdependence of activities, distributed knowledge, differentiated authority, temporal constraints, conflicting interests, and changing environmental conditions while the task is underway. Accordingly, we argue that organizational task difficulty constitutes a distinct and insufficiently recognized dimension of AI capability, and that existing approaches to benchmarking provide an incomplete basis for assessing how effectively AI agents can perform real-world organizational work. By drawing on organization theory, we develop a taxonomy of the structure of work that characterizes the conditions that agents face in organizational work. Applying this taxonomy to analyze existing benchmarks, we find limited and uneven coverage of organizational dimensions, with fewer than one quarter of prominent benchmarks claiming any of them. We therefore argue that AI benchmarks should increasingly move beyond the evaluation of technically difficult tasks toward the systematic assessment of agents' ability to navigate the organizational conditions under which economically valuable work takes place. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Computers and Society (cs.CY) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.03494 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T18:09:53.755Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).