Metraipolis › archive › dc_bf3fb7200f99

How Accurately Can the Energy Use of Spark Applications Be Estimated Based on Resource Utilisation? — arXiv:2608.09359

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.09359
License (as archived)
CC0-1.0
Archived
2026-10-19T08:52:41.204Z
SHA-256 of the archived text
2755bef1780fd089eca937f1761afb7aff2ddcb05a6e77555dcba357b0040293
Stable URL of this page
https://metraipolis.com/documento/dc_bf3fb7200f99

How to cite

How Accurately Can the Energy Use of Spark Applications Be Estimated Based on Resource Utilisation? — arXiv:2608.09359 — arXiv (en), https://arxiv.org/abs/2608.09359 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf3fb7200f99 (Metraipolis archive dc_bf3fb7200f99, sha256 2755bef1780fd089eca937f1761afb7aff2ddcb05a6e77555dcba357b0040293)
BibTeX
@misc{dcbf3fb7200f99,
  title = {How Accurately Can the Energy Use of Spark Applications Be Estimated Based on Resource Utilisation? — arXiv:2608.09359},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.09359},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf3fb7200f99; sha256 2755bef1780fd089eca937f1761afb7aff2ddcb05a6e77555dcba357b0040293},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.09359. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Distributed batch data processing applications are widely executed on cloud-based resources where restricted user access to node-level hardware energy counters hinders transparent sustainability accounting. Energy and carbon attribution methodologies therefore depend on power models and available resource utilisation traces, yet the accuracy of these estimates has to be validated while direct counters are available. In this work, we use Apache Spark running on Kubernetes as a case-study dataflow runtime and cluster resource manager to compare model-based energy estimates to Intel RAPL package and DRAM energy on an AWS bare-metal cloud and an on-premises cluster, comparing different CPU usage signals and memory coefficients. We show that external monitoring improves signed package-energy error relative to Spark task traces, reducing underestimation from -29.58% to -24.41% on AWS and from -24.00% to -16.22% on-premises.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-11 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2608.09359 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:50:23.754Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-08-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).