Metraipolis › archive › dc_bf5da4981c4e
Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling in Sub-2B Small Language Models for Edge Deployment — arXiv:2609.07370
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.07370
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T12:45:34.704Z
- SHA-256 of the archived text
6d36e3b4d9e4fd4b5f4790bf1d02095de6e4180ff51ae6159cbeae9b50f19642- Stable URL of this page
https://metraipolis.com/documento/dc_bf5da4981c4e
How to cite
Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling in Sub-2B Small Language Models for Edge Deployment — arXiv:2609.07370 — arXiv (en), https://arxiv.org/abs/2609.07370 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf5da4981c4e (Metraipolis archive dc_bf5da4981c4e, sha256 6d36e3b4d9e4fd4b5f4790bf1d02095de6e4180ff51ae6159cbeae9b50f19642)
BibTeX
@misc{dcbf5da4981c4e,
title = {Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling in Sub-2B Small Language Models for Edge Deployment — arXiv:2609.07370},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.07370},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf5da4981c4e; sha256 6d36e3b4d9e4fd4b5f4790bf1d02095de6e4180ff51ae6159cbeae9b50f19642},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.07370. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Resource constrained single-board computers including Raspberry Pi, NVIDIA Jetson Nano, Arduino UNO Q, Orange Pi, and LattePanda motivate on-device small language model (SLM) agents that reduce cloud dependence, improve data locality, and tolerate intermittent connectivity. Model Context Protocol (MCP)-style tool invocation demands more than fluent generation: an agent must emit machine-readable JSON, select the correct tool, supply all required arguments, and avoid unintended actions. We establish a platform-agnostic CPU baseline by evaluating five open-weight models below two billion parameters Phi-1.5, Pythia-1.4B, TinyLlama-1.1B-Chat, Qwen2.5-0.5B, and Qwen2.5-1.5B on 100 prompts spanning weather retrieval, web search, calculation, email composition, and task creation, under greedy decoding and nucleus sampling. A recovery parser strips Markdown fences, extracts brace-delimited substrings, and scores parseability, tool-name correctness, argument completeness, and value agreement. Under this criterion, Qwen2.5-1.5B achieves 75% (greedy) and 79% (sampling); Qwen2.5-0.5B achieves 72% (greedy) but drops to 32% under sampling. Phi-1.5 scores 0%; Pythia and TinyLlama reach at most 7%. A strict post-hoc audit finds only 5 of 1,000 raw responses directly parseable as JSON, exposing near-total dependence on output recovery. A CPU resource probe shows Qwen2.5-1.5B requires 7,960 MiB and 30.782 s mean latency; Qwen2.5-0.5B uses 3,637 MiB and 10.627 s, revealing a reliability-resource trade-off for edge deployment. These results do not cover the named boards directly or a full MCP implementation. Safe deployment requires schema validation, constrained generation, least-privilege execution, and human escalation for consequential actions. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.07370 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T12:30:25.116Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).