Metraipolis › archive › dc_5d4bb8504e8b

DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows — arXiv:2608.26546

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.26546
License (as archived)
CC0-1.0
Archived
2026-10-17T14:22:43.887Z
SHA-256 of the archived text
f7620f85b544903b36b05d2c89bea0286948cd45e669427ea460ba79e58fbed5
Stable URL of this page
https://metraipolis.com/documento/dc_5d4bb8504e8b

How to cite

DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows — arXiv:2608.26546 — arXiv (en), https://arxiv.org/abs/2608.26546 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5d4bb8504e8b (Metraipolis archive dc_5d4bb8504e8b, sha256 f7620f85b544903b36b05d2c89bea0286948cd45e669427ea460ba79e58fbed5)
BibTeX
@misc{dc5d4bb8504e8b,
  title = {DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows — arXiv:2608.26546},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.26546},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5d4bb8504e8b; sha256 f7620f85b544903b36b05d2c89bea0286948cd45e669427ea460ba79e58fbed5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26546. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Autonomous agents are increasingly adopted to complete complex, multi-tool workflows in real-world settings. However, existing benchmarks typically separate tasks by application or capability and evaluate agents in environments that are cleaner and more stable than those encountered in practice. We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions collected from a large-scale production agent platform. Each task preserves the relevant pre-solution interaction history, persistent configurations, and workspace state, and is then validated through human verification. The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories, with most tasks requiring multiple capability coordination. We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy, and assess performance using a hybrid deterministic and LLM-as-Judge evaluation protocol. Experiments across five representative autonomous-agent frameworks paired with four state-of-the-art LLMs reveal substantial gaps in strict task completion. Complementary robustness, efficiency, and diagnostic analyses further show that performance under environmental perturbations is jointly shaped by the capabilities of the LLM and the surrounding agent framework. The code and data are publicly available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.26546 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T13:16:53.501Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).