Metraipolis › archive › dc_7486bf1f9640

BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation — arXiv:2611.11761

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.11761
License (as archived)
CC0-1.0
Archived
2026-11-10T18:44:42.328Z
SHA-256 of the archived text
a8a02d7ba9481977c30e7b03e5fbf4f20150d84ae9a5ac25d491d4586799a112
Stable URL of this page
https://metraipolis.com/documento/dc_7486bf1f9640

How to cite

BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation — arXiv:2611.11761 — arXiv (en), https://arxiv.org/abs/2611.11761 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7486bf1f9640 (Metraipolis archive dc_7486bf1f9640, sha256 a8a02d7ba9481977c30e7b03e5fbf4f20150d84ae9a5ac25d491d4586799a112)
BibTeX
@misc{dc7486bf1f9640,
  title = {BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation — arXiv:2611.11761},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.11761},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7486bf1f9640; sha256 a8a02d7ba9481977c30e7b03e5fbf4f20150d84ae9a5ac25d491d4586799a112},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.11761. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM inference has two phases with different bottlenecks: LLM prefill runs once per request and is compute-intensive. Decoding repeats for every output token and is limited by memory bandwidth. To address the challenges associated with this issue, different solutions have been proposed. All of these solutions use the same model for both prefill and decode phases. We propose BridgeKV, which uses a larger model for prefill and a smaller model of the same family for decoding. A lightweight KV-stitching module translates the larger model's prompt cache into the smaller model's cache layout. It handles differences in cache shape and representation through learned local layer mixing and separate nonlinear maps for keys and values. With both models frozen, we train the module to align their caches and make the smaller model follow the larger model's next-token predictions. A second training stage adds low-rank adapters when further task adaptation is needed. Across six model pairs and five datasets, BridgeKV outperforms fine-tuned small-model baselines with matched trainable parameter budgets in all 30 settings. It closes up to 80% of the quality gap to using the larger model for both phases. On one A40 GPU, Llama-3.1-Instruct-8B to Llama-3.2-Instruct-1B achieves 3.0X faster requests than the larger model alone, with latency close to the smaller model alone. BridgeKV can improve the hardware and system solutions that separate prefill and decode phases across GPUs. A separate serving simulation models request arrivals, queues, and cache transfer. For Llama-3.1-Instruct-70B to Llama-3.2-Instruct-1B, it predicts up to 33.86X faster requests and 27.5X lower energy use than using the larger model for both prefill and decode phases.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.11761 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:37:52.443Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).