Metraipolis › archive › dc_d9d643e97a03

HoliBench: A Cross-Platform Benchmarking and Deployment Toolkit for Foundation Models in CPS-IoT Applications — arXiv:2609.12412

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.12412
License (as archived)
CC0-1.0
Archived
2026-10-16T03:08:36.319Z
SHA-256 of the archived text
97926c282b2cb730d0aa97eaad0d60a50ae1ed46674781fe76e1a9dc829dcf46
Stable URL of this page
https://metraipolis.com/documento/dc_d9d643e97a03

How to cite

HoliBench: A Cross-Platform Benchmarking and Deployment Toolkit for Foundation Models in CPS-IoT Applications — arXiv:2609.12412 — arXiv (en), https://arxiv.org/abs/2609.12412 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d9d643e97a03 (Metraipolis archive dc_d9d643e97a03, sha256 97926c282b2cb730d0aa97eaad0d60a50ae1ed46674781fe76e1a9dc829dcf46)
BibTeX
@misc{dcd9d643e97a03,
  title = {HoliBench: A Cross-Platform Benchmarking and Deployment Toolkit for Foundation Models in CPS-IoT Applications — arXiv:2609.12412},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.12412},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d9d643e97a03; sha256 97926c282b2cb730d0aa97eaad0d60a50ae1ed46674781fe76e1a9dc829dcf46},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.12412. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Foundation models, including large language models, vision-language models, and time-series foundation models, are increasingly deployed on embedded and edge platforms for CPS and IoT applications, where energy, latency, and memory are as critical as task accuracy. Existing benchmarking tools evaluate model capability in isolation, reporting accuracy assuming sufficient compute, while hardware profiling tools remain platform-specific and mutually incompatible. As a result, users lack a unified workflow for making deployment decisions across heterogeneous devices. We present HoliBench, a modular benchmarking and deployment toolkit that jointly characterizes accuracy, latency, and energy across platforms from single-board computers to GPU servers. Its platform abstraction layer calibrates cross-device measurement, and the toolkit supports multiple model modalities, inference engines, concurrencies, and existing evaluation harnesses. An interactive interface exposes constraint-aware configuration selection over a design space that is profiled once and reused across studies. Using HoliBench, we characterize 20 models across 7 device types, 3 quantization levels, 8 inference backends, and over 30 tasks, surfacing tradeoffs that existing tools miss: quantization reduces latency only on hardware with low-precision support, accuracy gains show diminishing returns relative to energy, and for autoregressive workloads, average inference power is approximately constant across output lengths. We further find that single-model profiles compose under sequential co-resident execution. In a multi-model CPS deployment, standalone profiles predict combined-pipeline latency and power within 1.2% and 2.5%, enabling deployment exploration without exhaustively profiling every pipeline configuration. We release HoliBench as open-source infrastructure for deployment-aware evaluation of foundation models.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-14 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Machine Learning (cs.LG); Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2609.12412 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T03:03:05.084Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).