Metraipolis › archive › dc_51ec878a9ef7
When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success — arXiv:2609.21187
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.21187
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-10T01:27:16.258Z
- SHA-256 of the archived text
742e2d5c66022fc174913701600b08df631468ac7455542fd74a0d08b1996b1c- Stable URL of this page
https://metraipolis.com/documento/dc_51ec878a9ef7
How to cite
When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success — arXiv:2609.21187 — arXiv (en), https://arxiv.org/abs/2609.21187 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_51ec878a9ef7 (Metraipolis archive dc_51ec878a9ef7, sha256 742e2d5c66022fc174913701600b08df631468ac7455542fd74a0d08b1996b1c)
BibTeX
@misc{dc51ec878a9ef7,
title = {When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success — arXiv:2609.21187},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.21187},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_51ec878a9ef7; sha256 742e2d5c66022fc174913701600b08df631468ac7455542fd74a0d08b1996b1c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21187. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Agent models are frequently evaluated one decision at a time, where the model predicts the next action based on the gold interaction history, which is scored against a reference. We investigate whether improvement under this protocol is predictive of improved autonomous workflow execution. We study pre-SFT and supervised fine-tuned (SFT) Qwen3 models at 4B and 14B parameters and Gemma 3 models at 4B and 12B parameters on multi-turn customer-support workflows. We find that SFT consistently improves text-turn success, and that overall next-turn success increases for every model under gold-history evaluation. However, these improvements do not transfer to autonomous workflow execution. Tool-specific gains also vary across metrics and models. None of the four SFT models succeeds under holistic workflow evaluation, with strict trajectory completion reaching at most 10.4% workflow success. Our results show that next-turn evaluation is not a reliable proxy for workflow success, motivating separate reporting of text quality, local action correctness, tool execution, and end-to-end task completion. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.21187 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-10T01:00:30.271Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).