Metraipolis › archive › dc_fed5ba2d4e43

BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference — arXiv:2609.13592

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.13592
License (as archived): CC0-1.0
Archived: 2026-10-15T23:17:35.783Z
SHA-256 of the archived text: 4ded2f33de5ebfbbb963049ab266bca02296d30e5ea1313196aa2d13b92c2c7d
Stable URL of this page: https://metraipolis.com/documento/dc_fed5ba2d4e43

How to cite

BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference — arXiv:2609.13592 — arXiv (en), https://arxiv.org/abs/2609.13592 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fed5ba2d4e43 (Metraipolis archive dc_fed5ba2d4e43, sha256 4ded2f33de5ebfbbb963049ab266bca02296d30e5ea1313196aa2d13b92c2c7d)

BibTeX

@misc{dcfed5ba2d4e43,
  title = {BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference — arXiv:2609.13592},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.13592},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fed5ba2d4e43; sha256 4ded2f33de5ebfbbb963049ab266bca02296d30e5ea1313196aa2d13b92c2c7d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.13592. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

GPU memory bandwidth and capacity limit throughput in large language model (LLM) inference. The GPU memory system consists of a primary tier of high-bandwidth memory (HBM) and a secondary tier of host memory connected via CPU-to-GPU interconnect. Current serving systems treat the tiers hierarchically: they serve exclusively from HBM when data fits, and otherwise prefetch data from host memory to HBM before use. In both cases, the host memory bandwidth is never well utilized. Prefetching expands capacity by utilizing host memory, but consumes HBM bandwidth for writes, reducing the bandwidth available for demand loads. We observe that fully utilizing both host and HBM bandwidth requires each wave of GPU threadblocks to access both tiers concurrently and in proportion to their bandwidth ratio. Existing bandwidth-proportional placement strategies fail to provide concurrency because they are not aware of GPU waves, and the large 2MB GPU page size. This paper presents BOOST, the first runtime system that provides concurrent and proportional access to both GPU memory tiers, extracting the combined bandwidth of host memory and HBM for LLM inference without kernel changes. The key insight in BOOST is to use kernel access patterns to make page allocation and runtime data management wave-aware. For static model weights, BOOST applies modulo-based page placement that eliminates access-ratio variance; for dynamically provisioned attention key-value (KV) pairs, it makes the free KV page pool wave-aware. We integrate BOOST into vLLM and evaluate on a Grace Hopper system. At iso-batch size, BOOST improves Time-per-Output-Token (TPOT) by 4.3% over HBM-only serving, whereas prefetching degrades TPOT by 6%. In high-throughput serving, BOOST improves throughput by 31% on average, outperforming prefetching by 15%.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-15 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Hardware Architecture (cs.AR); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.13592 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T22:54:12.954Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).