Metraipolis › archive › dc_de1ca7d9a8bf

DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments? — arXiv:2608.10366

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.10366
License (as archived)
CC0-1.0
Archived
2026-10-19T06:08:19.948Z
SHA-256 of the archived text
13a31ba23418cddcfdbef52d68a7ea02070d1469f743f62e03d98d392554e115
Stable URL of this page
https://metraipolis.com/documento/dc_de1ca7d9a8bf

How to cite

DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments? — arXiv:2608.10366 — arXiv (en), https://arxiv.org/abs/2608.10366 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_de1ca7d9a8bf (Metraipolis archive dc_de1ca7d9a8bf, sha256 13a31ba23418cddcfdbef52d68a7ea02070d1469f743f62e03d98d392554e115)
BibTeX
@misc{dcde1ca7d9a8bf,
  title = {DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments? — arXiv:2608.10366},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.10366},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_de1ca7d9a8bf; sha256 13a31ba23418cddcfdbef52d68a7ea02070d1469f743f62e03d98d392554e115},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.10366. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments. Yet existing benchmarks lack real-computer interaction and do not evaluate whether agents can execute complete end-to-end data-science workflows in realistic computing environments, failing to capture the multi-stage, multi-tool nature of data-science practice. We introduce DSAgentBench, the first benchmark to evaluate whether agents can automate full data-science workflows inside real computer environments. DSAgentBench contains 275 diverse tasks covering the entire data-science life-cycle, reflecting the complexity and tool coordination required in practice. Each task requires grounding decisions in intermediate outputs and coordinated tool use, and includes a deterministic evaluator that verifies analytical correctness, visual outputs, and model performance rather than code-only execution. Our extensive experiments with 15 closed- and open-source models show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success, while all open-source agents remain below 1%, frequently failing at tool orchestration, OS grounding, and multi-step reasoning. These results reveal a substantial capability gap between current agentic systems and real data-science workflows, positioning DSAgentBench as a foundation for developing grounded, verifiable, autonomous data-science agents. We release DSAgentBench at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-12 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.10366 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:57:28.260Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).