Metraipolis › archive › dc_bee05942efbf

BasisKV: Value Latents for Communication and Key Routing — arXiv:2611.02583

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.02583
License (as archived): CC0-1.0
Archived: 2026-11-03T10:43:54.549Z
SHA-256 of the archived text: a1e8eaf931b0aff0c0e76155d9371f8a7db924bfce9c9aabf2d16ebacfc521d9
Stable URL of this page: https://metraipolis.com/documento/dc_bee05942efbf

How to cite

BasisKV: Value Latents for Communication and Key Routing — arXiv:2611.02583 — arXiv (en), https://arxiv.org/abs/2611.02583 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bee05942efbf (Metraipolis archive dc_bee05942efbf, sha256 a1e8eaf931b0aff0c0e76155d9371f8a7db924bfce9c9aabf2d16ebacfc521d9)

BibTeX

@misc{dcbee05942efbf,
  title = {BasisKV: Value Latents for Communication and Key Routing — arXiv:2611.02583},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02583},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bee05942efbf; sha256 a1e8eaf931b0aff0c0e76155d9371f8a7db924bfce9c9aabf2d16ebacfc521d9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02583. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Long-context large language model (LLM) inference is increasingly constrained by the cost of maintaining an ever-growing KV-cache. Low-rank K/V projections provide a promising way to compress the KV-cache with minimal quality loss, but prior work has largely optimized KV-cache compression in isolation, leaving distributed inference, especially with tensor parallelism where attention computation is partitioned across multiple GPUs, largely underexplored. We make two observations: (i) tensor-parallel communication is an important cost when deploying low-rank KV-cache compression in distributed inference; and (ii) Value compression can be associated with the output projection, allowing the same low-rank Value representation to reduce both Value-cache width and tensor-parallel communication. Based on these observations, we propose BasisKV, a low-rank KV-cache compression method for communication-intensive, long-context LLM inference. BasisKV jointly optimizes the low-rank factors across per-GPU attention partitions against the complete attention output, rather than fitting each partition in isolation. The compressed Value cache is further reused to help identify which Key pages should be accessed: it predicts part of the Key information needed for routing, while a Key-specific residual stores what cannot be recovered from Value. The original Keys are then retrieved only for the selected pages to compute exact QK scores. Across multiple LLM families and system settings, BasisKV achieves a better quality--efficiency tradeoff than existing low-rank baselines, reaching up to 1.42x end-to-end speedup under 8-GPU tensor-parallel inference. Our code is available at this https URL

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.02583 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:39:50.289Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).