Metraipolis › archive › dc_9ae96ab48cfe

HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management — arXiv:2608.07009

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.07009
License (as archived)
CC0-1.0
Archived
2026-10-19T09:47:48.800Z
SHA-256 of the archived text
6e8bb8897e162412f65e0c1b14874b3003415d3b93c9a4ec8f627ca442f9cd36
Stable URL of this page
https://metraipolis.com/documento/dc_9ae96ab48cfe

How to cite

HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management — arXiv:2608.07009 — arXiv (en), https://arxiv.org/abs/2608.07009 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9ae96ab48cfe (Metraipolis archive dc_9ae96ab48cfe, sha256 6e8bb8897e162412f65e0c1b14874b3003415d3b93c9a4ec8f627ca442f9cd36)
BibTeX
@misc{dc9ae96ab48cfe,
  title = {HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management — arXiv:2608.07009},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.07009},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9ae96ab48cfe; sha256 6e8bb8897e162412f65e0c1b14874b3003415d3b93c9a4ec8f627ca442f9cd36},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.07009. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Top-k sparse attention makes long-context LLM decoding cheap to compute: each step reads only a few thousand selected KV entries rather than the full context. Serving systems, however, typically keep the entire KV cache in GPU HBM so that every position stays selectable, so a request's memory bill still grows with its full context length--decoding hits a capacity wall long before it runs out of compute, and a context whose KV cache exceeds HBM cannot be served at all. We present HiSparse, an exact, indexer-agnostic hierarchical KV cache for sparse-attention serving. HiSparse keeps each request's full KV history in host memory and bounds its decode footprint with a small, fixed-size GPU cache; a fused CUDA kernel resolves each layer's selections--hit detection, LRU replacement, and host-to-device fetches--inside the decode CUDA graph; and, for models that share selections across layers, exact layer-wise prefetching hides roughly half of the remaining miss overhead. Because only KV placement changes, model outputs are unchanged. HiSparse is merged into upstream SGLang and evaluated across three sparse-attention families (DSA, NSA, and Quest) on H200, B200, and GH200 platforms: it improves peak generation throughput by up to 4.7x on long-context workloads while preserving comparable per-token latency and reducing time-to-first-token at high load--and a no-IO oracle shows the resolution mechanism itself adds no measurable per-token cost, leaving host-device IO as the only price of bounded residency.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-10 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2608.07009 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T09:47:05.173Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-08-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).