Metraipolis › archive › dc_fe0bc959768c
Introducing Cluster Efficiency to LLM Serving: Pooling Economical Accelerators over Fast Interconnects — arXiv:2611.02975
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02975
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:12:44.998Z
- SHA-256 of the archived text
005707af740a1adaa30550fe86e80e92fd5abda8188ea8feca2553d934b7b55a- Stable URL of this page
https://metraipolis.com/documento/dc_fe0bc959768c
How to cite
Introducing Cluster Efficiency to LLM Serving: Pooling Economical Accelerators over Fast Interconnects — arXiv:2611.02975 — arXiv (en), https://arxiv.org/abs/2611.02975 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fe0bc959768c (Metraipolis archive dc_fe0bc959768c, sha256 005707af740a1adaa30550fe86e80e92fd5abda8188ea8feca2553d934b7b55a)
BibTeX
@misc{dcfe0bc959768c,
title = {Introducing Cluster Efficiency to LLM Serving: Pooling Economical Accelerators over Fast Interconnects — arXiv:2611.02975},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02975},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fe0bc959768c; sha256 005707af740a1adaa30550fe86e80e92fd5abda8188ea8feca2553d934b7b55a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02975. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Long-context large language model (LLM) serving places substantial demands on compute, memory capacity, and memory bandwidth. Economical accelerators can offer these resources at lower cost, but their limited per-device capacity requires pooling across devices, making communication overhead critical. We study cluster efficiency, measured in tokens per dollar and per joule, and show how low-latency, high-bandwidth interconnects enable economical accelerators to retain their cost advantage. Guided by a characterization of coding-agent workloads and accelerator scaling, we realize this design on the Tenstorrent Galaxy, a cluster of Blackhole chips. The serving stack combines integrated communication engines, pipelined ring transfers, communication-computation overlap, and device-side decode replay to reduce collective overhead. Across three dense models in BF16 and 8-bit configurations with 8K-256K-token contexts, the Galaxy achieves 1.07-2.24$\times$ the tokens per dollar of an eight-GPU RTX PRO 6000 Blackwell server under a three-year acquisition-and-electricity cost model. Emulating TCP-class communication on the same chips reduces throughput by 23.5-67.4% and eliminates the cost advantage in every evaluated configuration. These results demonstrate how fast interconnects translate the per-dollar resource advantages of economical accelerators into cost-efficient long-context LLM serving. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2611.02975 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T11:00:07.165Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).