Metraipolis › archive › dc_d009fee5c32f
Balancing Workload Performance and Slurm Stress: Four Nextflow Deployment Strategies — arXiv:2608.13824
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.13824
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T02:28:02.822Z
- SHA-256 of the archived text
885909d181ae8e67631a6a89e2c0b4a00b5bd4b3c71c9e98af9415adcc6856a1- Stable URL of this page
https://metraipolis.com/documento/dc_d009fee5c32f
How to cite
Balancing Workload Performance and Slurm Stress: Four Nextflow Deployment Strategies — arXiv:2608.13824 — arXiv (en), https://arxiv.org/abs/2608.13824 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d009fee5c32f (Metraipolis archive dc_d009fee5c32f, sha256 885909d181ae8e67631a6a89e2c0b4a00b5bd4b3c71c9e98af9415adcc6856a1)
BibTeX
@misc{dcd009fee5c32f,
title = {Balancing Workload Performance and Slurm Stress: Four Nextflow Deployment Strategies — arXiv:2608.13824},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.13824},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d009fee5c32f; sha256 885909d181ae8e67631a6a89e2c0b4a00b5bd4b3c71c9e98af9415adcc6856a1},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.13824. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Wide Nextflow fan-outs on shared Slurm clusters can submit tens of thousands of short tasks. Deployment settings route them through individual jobs, arrays, or nested schedulers inside enclosing allocations. These settings determine workflow turnaround and RPC volume, a shared cost that can degrade scheduler responsiveness. Existing comparisons evaluate whole workflow systems, while per-task queueing metrics cannot span architectures that dispatch inside existing allocations. We contribute a reproducible measurement protocol and benchmark harness. A clean-start clock begins before backend startup or allocation requests, placing architecturally different backends on a common time axis. Per-user Slurm sdiag counters attribute request count as the primary RPC demand measure and controller processing time as sensitivity context, separate from cluster-wide state. We apply the method to Slurm native dispatch, Slurm job arrays, HyperQueue, and Flux on the shared ASU Phoenix production cluster and a single-user Dev cluster. On Phoenix, every aggregation strategy improves both objectives relative to native dispatch; Flux has the lowest RPC demand, while HyperQueue's fastest median walltime is not stable across replicates. On Dev, arrays and Flux improve walltime, while HyperQueue trades the lowest RPC demand for the slowest completion. The method lets an HPC site compare deployment strategies using both user-visible performance and scheduler impact, then select the fastest strategy within its own RPC-demand limit. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-17 in cs.DC). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2608.13824 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T02:20:49.502Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).