Metraipolis › archive › dc_8739716e3e2e

ShowTellArena: Evaluating Business Workflow Understanding from Demonstrations — arXiv:2609.25467

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.25467
License (as archived)
CC0-1.0
Archived
2026-10-08T13:45:00.741Z
SHA-256 of the archived text
a7113788de5ad7f8094ed0f313b7f1928a04c6091fd112789e4be02a47897db7
Stable URL of this page
https://metraipolis.com/documento/dc_8739716e3e2e

How to cite

ShowTellArena: Evaluating Business Workflow Understanding from Demonstrations — arXiv:2609.25467 — arXiv (en), https://arxiv.org/abs/2609.25467 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8739716e3e2e (Metraipolis archive dc_8739716e3e2e, sha256 a7113788de5ad7f8094ed0f313b7f1928a04c6091fd112789e4be02a47897db7)
BibTeX
@misc{dc8739716e3e2e,
  title = {ShowTellArena: Evaluating Business Workflow Understanding from Demonstrations — arXiv:2609.25467},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.25467},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8739716e3e2e; sha256 a7113788de5ad7f8094ed0f313b7f1928a04c6091fd112789e4be02a47897db7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.25467. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We often teach a colleague by showing the work and explaining the decisions as we go. How can we check what an agent understood from the same lesson? We introduce ShowTellArena, a benchmark protocol and public dataset for comprehension after narrated business demonstrations. The v1.0 release contains 50 business workflow tasks, with recordings, screenshots, narration, fixture seeds, and 502 questions. Tasks span finance, hiring, procurement, customer decisions, inventory, and logistics. The protocol holds the business scenario and quiz fixed while allowing each product to capture the lesson through its own teaching interface. Questions test operational rules, boundaries, exceptions, and errors in proposed automations. We analyze 218 selected pilot attempts across 39 workflow cases, including 28 cases attempted by all three evaluated systems. These exploratory results expose both answer errors and failures to complete the teaching experience. We describe the release's verification gaps and the pilot's uneven coverage, exclusions, and grading provenance. The contribution is an inspectable dataset and assessment workflow that others can extend; the selected pilot is not a controlled product ranking.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-23 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.25467 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T13:18:38.084Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).