Metraipolis › archive › dc_86a18a601bae
CAREFUL-Bench: Effect-Aware, Judge-Free Evaluation of Multi-Turn LLM Function Calling — arXiv:2611.02978
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02978
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T10:44:11.960Z
- SHA-256 of the archived text
ed113f29204198482680f230e82dd3fa73c9a368d418bec5bd2f27ad4f7cc544- Stable URL of this page
https://metraipolis.com/documento/dc_86a18a601bae
How to cite
CAREFUL-Bench: Effect-Aware, Judge-Free Evaluation of Multi-Turn LLM Function Calling — arXiv:2611.02978 — arXiv (en), https://arxiv.org/abs/2611.02978 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_86a18a601bae (Metraipolis archive dc_86a18a601bae, sha256 ed113f29204198482680f230e82dd3fa73c9a368d418bec5bd2f27ad4f7cc544)
BibTeX
@misc{dc86a18a601bae,
title = {CAREFUL-Bench: Effect-Aware, Judge-Free Evaluation of Multi-Turn LLM Function Calling — arXiv:2611.02978},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02978},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_86a18a601bae; sha256 ed113f29204198482680f230e82dd3fa73c9a368d418bec5bd2f27ad4f7cc544},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02978. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The multi-turn checker of the Berkeley Function Calling Leaderboard (BFCL) evaluates final state correctness but ignores the effects of intermediate calls: extra calls are free, function documentation does not describe side effects, and the scripted user never responds to proposals. Consequently, models asking for confirmation before destructive actions are penalized; 260 of 734 reference turns in the base category perform destructive actions without confirmation. We make BFCL effect-aware without replacing it. We infer effect declarations for all 128 functions by executing them in BFCL simulators and comparing state changes, with destructive writes identified by three annotators. A judge-free replay scorer reproduces official results across all 800 reference trajectories while reporting unrequested destructive calls, duplicate writes, redundant reads, surplus steps, and their token costs. We also introduce a confirmation-aware protocol with a deterministic simulated user. Across 11 open models, effect-aware scoring preserves the official ranking but reveals substantial differences in execution quality: the leading model loses 14.0 points due to unrequested destructive calls, while surplus steps consume up to 28% of input tokens. Adding effect hints to function documentation has no impact on eight evaluated models. Allowing the simulated user to respond recovers up to four points for models that seek confirmation. However, the best model completes only six of 576 conversations requiring destructive actions with explicit confirmation. These results expose a key limitation of current function-calling benchmarks: they cannot distinguish between asking for clarification and obtaining permission before potentially destructive actions. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2611.02978 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T10:39:50.327Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).