Metraipolis › archive › dc_aea79b09634f
CAREFUL-Bench: Effect-Aware, Judge-Free Evaluation of Multi-Turn LLM Function Calling — arXiv:2611.02978
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02978
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T10:33:37.043Z
- SHA-256 of the archived text
39434675113bfaa2e740601bef097f1ee9b0904b98bde40854d07926adf31a3f- Stable URL of this page
https://metraipolis.com/documento/dc_aea79b09634f
How to cite
CAREFUL-Bench: Effect-Aware, Judge-Free Evaluation of Multi-Turn LLM Function Calling — arXiv:2611.02978 — arXiv (en), https://arxiv.org/abs/2611.02978 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aea79b09634f (Metraipolis archive dc_aea79b09634f, sha256 39434675113bfaa2e740601bef097f1ee9b0904b98bde40854d07926adf31a3f)
BibTeX
@misc{dcaea79b09634f,
title = {CAREFUL-Bench: Effect-Aware, Judge-Free Evaluation of Multi-Turn LLM Function Calling — arXiv:2611.02978},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02978},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aea79b09634f; sha256 39434675113bfaa2e740601bef097f1ee9b0904b98bde40854d07926adf31a3f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02978. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The multi-turn checker of the Berkeley Function Calling Leaderboard (BFCL) evaluates final state correctness but ignores the effects of intermediate calls: extra calls are free, function documentation does not describe side effects, and the scripted user never responds to proposals. Consequently, models asking for confirmation before destructive actions are penalized; 260 of 734 reference turns in the base category perform destructive actions without confirmation. We make BFCL effect-aware without replacing it. We infer effect declarations for all 128 functions by executing them in BFCL simulators and comparing state changes, with destructive writes identified by three annotators. A judge-free replay scorer reproduces official results across all 800 reference trajectories while reporting unrequested destructive calls, duplicate writes, redundant reads, surplus steps, and their token costs. We also introduce a confirmation-aware protocol with a deterministic simulated user. Across 11 open models, effect-aware scoring preserves the official ranking but reveals substantial differences in execution quality: the leading model loses 14.0 points due to unrequested destructive calls, while surplus steps consume up to 28% of input tokens. Adding effect hints to function documentation has no impact on eight evaluated models. Allowing the simulated user to respond recovers up to four points for models that seek confirmation. However, the best model completes only six of 576 conversations requiring destructive actions with explicit confirmation. These results expose a key limitation of current function-calling benchmarks: they cannot distinguish between asking for clarification and obtaining permission before potentially destructive actions. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2611.02978 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T10:29:48.613Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).