Metraipolis › archive › dc_f79689af56ff

One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows — arXiv:2608.19741

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.19741
License (as archived)
CC0-1.0
Archived
2026-10-18T11:39:14.015Z
SHA-256 of the archived text
ad25aae61c56dd796788b74264e0071f7c0583278693bca05fa2a7a087564fde
Stable URL of this page
https://metraipolis.com/documento/dc_f79689af56ff

How to cite

One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows — arXiv:2608.19741 — arXiv (en), https://arxiv.org/abs/2608.19741 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f79689af56ff (Metraipolis archive dc_f79689af56ff, sha256 ad25aae61c56dd796788b74264e0071f7c0583278693bca05fa2a7a087564fde)
BibTeX
@misc{dcf79689af56ff,
  title = {One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows — arXiv:2608.19741},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.19741},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f79689af56ff; sha256 ad25aae61c56dd796788b74264e0071f7c0583278693bca05fa2a7a087564fde},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.19741. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call: agents must gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realize the correct persistent state transition without collateral effects. In this paper, we introduce Thinkingbox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state. Built on this sandbox, Thinkingbox-bench contains 507 policy-conditioned workflows across business scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt is evaluated by task-specific executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects; designated tasks additionally check required properties of the final response. Our experiments reveal that even the strongest proprietary and open-weight models show steep reliability drops: Claude Opus 5 falls from 66.50% pass@1 to 47.53% pass^20, and Kimi-K3 from 57.37% pass@1 to 17.60% pass^20. Moreover, many failed trials terminate cleanly after valid state-changing actions, so response- or tool-call-level signals poorly proxy end-to-end completion. Thinkingbox-bench reveals a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks. We release both Thinkingbox ( this https URL ) and Thinkingbox-bench ( this https URL ).

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-21 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Databases (cs.DB).
Pagina del lavoro: https://arxiv.org/abs/2608.19741 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T11:38:27.905Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).