Metraipolis › archive › dc_73732ae26d52
Evolving Self-tuning Benchmarks for Long-Horizon Tool Use — arXiv:2610.27307
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.27307
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-21T07:06:28.336Z
- SHA-256 of the archived text
212029a8c926e6b8062c6c3b8263c3bcae2a623213deeed9655e7723ed54f689- Stable URL of this page
https://metraipolis.com/documento/dc_73732ae26d52
How to cite
Evolving Self-tuning Benchmarks for Long-Horizon Tool Use — arXiv:2610.27307 — arXiv (en), https://arxiv.org/abs/2610.27307 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_73732ae26d52 (Metraipolis archive dc_73732ae26d52, sha256 212029a8c926e6b8062c6c3b8263c3bcae2a623213deeed9655e7723ed54f689)
BibTeX
@misc{dc73732ae26d52,
title = {Evolving Self-tuning Benchmarks for Long-Horizon Tool Use — arXiv:2610.27307},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.27307},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_73732ae26d52; sha256 212029a8c926e6b8062c6c3b8263c3bcae2a623213deeed9655e7723ed54f689},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.27307. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
For AI assistants to move beyond simple chat interfaces and reliably assist users with applications like Gmail, Venmo, and Slack, we need to improve their tool-use capabilities. However, existing tool use benchmarks are still limited and difficult to scale because they rely on human supervision or are too domain-specific. To make data generation scalable, we propose, TURTLE (**T**ool **U**se **R**elated **T**asks with **L**ong-Horizon **E**volution), an Unsupervised Environment Design (UED) approach to evolve existing tasks using a proposer, a solver, and a feedback generator large language model agents. The proposer is rewarded for difficult yet solvable tasks based on the solver's empirical performance, and the feedback generator summarizes the solver's execution traces for the proposer to exploit. TURTLE produces evolving, self-tuning test suites adaptable to different environment requirements, from supporting user simulators and dense document retrieval to interacting across multiple apps. Empirical results on AppWorld, $\tau$-Bench, and AutomationBench using Gemma 4 31B-it show that the evolved tasks target the desired difficulty range (e.g., a 165\% increase in the percentage of hard tasks on AppWorld), increase task diversity (in terms of reasoning and simultaneous tool interactions), and elicit a broader range of failure modes than the original tasks. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.27307 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-21T07:05:43.395Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).