Metraipolis › archive › dc_61600599be66
Introducing Cluster Efficiency to LLM Serving: Pooling Economical Accelerators over Fast Interconnects — arXiv:2611.02975
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02975
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:26:45.560Z
- SHA-256 of the archived text
9b34f8ae87c41ac6830fbfbfdfd34308def1f2bcbadfc8d74211a90b1f0a50b9- Stable URL of this page
https://metraipolis.com/documento/dc_61600599be66
How to cite
Introducing Cluster Efficiency to LLM Serving: Pooling Economical Accelerators over Fast Interconnects — arXiv:2611.02975 — arXiv (en), https://arxiv.org/abs/2611.02975 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_61600599be66 (Metraipolis archive dc_61600599be66, sha256 9b34f8ae87c41ac6830fbfbfdfd34308def1f2bcbadfc8d74211a90b1f0a50b9)
BibTeX
@misc{dc61600599be66,
title = {Introducing Cluster Efficiency to LLM Serving: Pooling Economical Accelerators over Fast Interconnects — arXiv:2611.02975},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02975},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_61600599be66; sha256 9b34f8ae87c41ac6830fbfbfdfd34308def1f2bcbadfc8d74211a90b1f0a50b9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02975. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Long-context large language model (LLM) serving places substantial demands on compute, memory capacity, and memory bandwidth. Economical accelerators can offer these resources at lower cost, but their limited per-device capacity requires pooling across devices, making communication overhead critical. We study cluster efficiency, measured in tokens per dollar and per joule, and show how low-latency, high-bandwidth interconnects enable economical accelerators to retain their cost advantage. Guided by a characterization of coding-agent workloads and accelerator scaling, we realize this design on the Tenstorrent Galaxy, a cluster of Blackhole chips. The serving stack combines integrated communication engines, pipelined ring transfers, communication-computation overlap, and device-side decode replay to reduce collective overhead. Across three dense models in BF16 and 8-bit configurations with 8K-256K-token contexts, the Galaxy achieves 1.07-2.24$\times$ the tokens per dollar of an eight-GPU RTX PRO 6000 Blackwell server under a three-year acquisition-and-electricity cost model. Emulating TCP-class communication on the same chips reduces throughput by 23.5-67.4% and eliminates the cost advantage in every evaluated configuration. These results demonstrate how fast interconnects translate the per-dollar resource advantages of economical accelerators into cost-efficient long-context LLM serving. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2611.02975 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T18:10:22.841Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).