Metraipolis › archive › dc_f5e462ebf506

TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching — arXiv:2609.21172

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.21172
License (as archived)
CC0-1.0
Archived
2026-10-10T01:44:50.569Z
SHA-256 of the archived text
52047fd14d17d00f3f2d9364061baa81d087ee108b41aec81df44f3af5f37552
Stable URL of this page
https://metraipolis.com/documento/dc_f5e462ebf506

How to cite

TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching — arXiv:2609.21172 — arXiv (en), https://arxiv.org/abs/2609.21172 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f5e462ebf506 (Metraipolis archive dc_f5e462ebf506, sha256 52047fd14d17d00f3f2d9364061baa81d087ee108b41aec81df44f3af5f37552)
BibTeX
@misc{dcf5e462ebf506,
  title = {TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching — arXiv:2609.21172},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.21172},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f5e462ebf506; sha256 52047fd14d17d00f3f2d9364061baa81d087ee108b41aec81df44f3af5f37552},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21172. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) are moving onto mobile devices for increasingly diverse workloads over text, images, video, and audio. These applications often require long contexts, making the Key-Value (KV) cache a dominant memory bottleneck because it grows linearly with sequence length and is accessed at every decoding step. Prior work reduces KV-cache footprint through low-rank compression, token eviction, or flash offloading, but the resulting reconstruction overhead, irreversible token loss, or I/O stalls can offset the benefit of saving memory. We present TierKV, a mobile LLM inference framework built on Predictive Multi-Tier Cache Optimization (PMCO). Before decoding starts, PMCO predicts future cache demand from prefill hidden states and jointly assigns tokens to exact, low-rank, and flash-offloaded tiers under the device memory and accuracy budgets. This formulation retains access to the full context, removes the circular dependency of reactive eviction, and admits a closed-form solver that selects tier boundaries and per-layer ranks at runtime. Across eight text, vision, and audio models on three mobile SoCs, TierKV improves prefill throughput by up to 17.6x over existing mobile LLM frameworks, reduces RAM-resident KV cache by 12.5-34%, thereby enabling substantially longer contexts under the same memory budget, while incurring only minor accuracy degradation.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Systems and Control (eess.SY).
Pagina del lavoro: https://arxiv.org/abs/2609.21172 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T01:14:16.615Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).