Metraipolis › archive › dc_3e3512bf4aa3

Fairway: Load-Aware KV Cache Reuse for LLM Prefill Scheduling — arXiv:2611.02082

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.02082
License (as archived)
CC0-1.0
Archived
2026-11-03T18:26:41.236Z
SHA-256 of the archived text
d6cc901565b9cefccfcfdb0a7f0bb9be1b0f89a6e9e489b67b085635bf6413f6
Stable URL of this page
https://metraipolis.com/documento/dc_3e3512bf4aa3

How to cite

Fairway: Load-Aware KV Cache Reuse for LLM Prefill Scheduling — arXiv:2611.02082 — arXiv (en), https://arxiv.org/abs/2611.02082 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3e3512bf4aa3 (Metraipolis archive dc_3e3512bf4aa3, sha256 d6cc901565b9cefccfcfdb0a7f0bb9be1b0f89a6e9e489b67b085635bf6413f6)
BibTeX
@misc{dc3e3512bf4aa3,
  title = {Fairway: Load-Aware KV Cache Reuse for LLM Prefill Scheduling — arXiv:2611.02082},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02082},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3e3512bf4aa3; sha256 d6cc901565b9cefccfcfdb0a7f0bb9be1b0f89a6e9e489b67b085635bf6413f6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02082. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM-based applications often employ agentic workflows and retrieval-augmented generation (RAG), which build long prompts from shared context. When prefixes are shared across requests, serving engines reuse the key-value (KV) cache to skip parts of the prefill computation. Reuse is typically graded: the uncached work of requests in a queue can differ by up to 100x. In this setting, the order of scheduling determines the time-to-first-token (TTFT) for the queued requests. KV reuse-first scheduling, which prioritizes the requests with the longest KV cache hit, gives the lowest median TTFT but starves expensive requests under load spikes; first-come-first-served (FCFS) scheduling bounds the tail but raises the median TTFT. We present Fairway, a prefill scheduler built on adaptive reuse prioritization: it favors KV cache reuse while queues are short, and shifts toward fair allocation as requests wait longer relative to their own prefill time. Fairway employs a virtual dual lane model: a KV reuse-first Fast lane and a Fair lane atop the serving engine's waiting queue that enables speeding up of cheap reuse requests without starving expensive ones. This model automatically adapts to the workload without any per-workload tuning. Fairway lowers the tail TTFT at no cost to the median in transient overload. Across workloads and load levels, Fairway matches or beats the best baseline, with a p99 TTFT up to 8.6x lower than prior prefix-aware schedulers. During load spikes, it keeps the tail of FCFS at a 6x lower median TTFT.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2611.02082 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:10:22.826Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).