Metraipolis › archive › dc_e5f586004bc3

Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Serving — arXiv:2609.25782

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.25782
License (as archived)
CC0-1.0
Archived
2026-10-08T14:37:38.671Z
SHA-256 of the archived text
9d57df07b75295d5b3e073f2bb3e2386cb86fa23b7de5c5c9adb323e9f69c634
Stable URL of this page
https://metraipolis.com/documento/dc_e5f586004bc3

How to cite

Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Serving — arXiv:2609.25782 — arXiv (en), https://arxiv.org/abs/2609.25782 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e5f586004bc3 (Metraipolis archive dc_e5f586004bc3, sha256 9d57df07b75295d5b3e073f2bb3e2386cb86fa23b7de5c5c9adb323e9f69c634)
BibTeX
@misc{dce5f586004bc3,
  title = {Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Serving — arXiv:2609.25782},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.25782},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e5f586004bc3; sha256 9d57df07b75295d5b3e073f2bb3e2386cb86fa23b7de5c5c9adb323e9f69c634},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.25782. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model (LLM) serving is increasingly agentic, with multi-turn sessions that idle between actions yet must retain their full context. Limited GPU memory capacity forces inactive KV states to be evicted, so resuming a session incurs either costly recomputation or slow interconnect transfers. To address this, high bandwidth flash (HBF)-an on-package 3D-NAND memory offering orders-of-magnitude greater capacity than high bandwidth memory (HBM) at comparable read bandwidth-has emerged as a strong candidate. However, its high read energy and limited write endurance make it impractical to serve all KV traffic. Fortunately, our analysis shows that agentic KV states exhibit distinct access patterns: a small hot set is read for every decoding step, while a large cold pool is read only when a paused session resumes. Exploiting this, we place the hot set in HBM and the cold pool in HBF, forming a hot-cold KV hierarchy within the GPU memory tier. On agentic workloads with Qwen3-Coder-30B-A3B, our design delivers 14 ms time-between-tokens (TBT) and adds only $\approx$0.1 ms of resume latency on top of prefill, while hosting $24\times$ more concurrent sessions per GPU. By confining HBM to the hot set, our design also cuts read power by 7.6 kW per 8-GPU node relative to serving all KV from flash-establishing HBF as a cold-tier complement to HBM rather than its replacement.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-23 in cs.AR).
Categorie arXiv: Hardware Architecture (cs.AR).
Pagina del lavoro: https://arxiv.org/abs/2609.25782 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T13:47:52.523Z dalla pagina https://arxiv.org/catchup/cs.AR/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).