Metraipolis › archive › dc_84d82efcf8e1

Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management — arXiv:2609.06940

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.06940
License (as archived)
CC0-1.0
Archived
2026-10-16T13:07:13.758Z
SHA-256 of the archived text
d773d8275346561cfe054b3c760043fa9f89d27540d9f999012cfb8d1b222d71
Stable URL of this page
https://metraipolis.com/documento/dc_84d82efcf8e1

How to cite

Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management — arXiv:2609.06940 — arXiv (en), https://arxiv.org/abs/2609.06940 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_84d82efcf8e1 (Metraipolis archive dc_84d82efcf8e1, sha256 d773d8275346561cfe054b3c760043fa9f89d27540d9f999012cfb8d1b222d71)
BibTeX
@misc{dc84d82efcf8e1,
  title = {Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management — arXiv:2609.06940},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.06940},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_84d82efcf8e1; sha256 d773d8275346561cfe054b3c760043fa9f89d27540d9f999012cfb8d1b222d71},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.06940. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model (LLM) inference increasingly spans models that differ in size, capability, price, and provider. This shift creates two costs for developers. One is the integration cost of choosing among and switching between many models. The other is the inference cost of rebuilding a KV cache when it is unavailable or incompatible with the selected model. We define and analyze the Unified AI Gateway as a system setting for an edge-deployed AI traffic hub. It coordinates model routing, KV cache management, and compute placement across end devices, edge resources, and cloud model services. At request time, the gateway jointly selects a target model, an execution site, and a KV cache action under task-quality, latency, cost, and resource constraints. In parallel, background cache-management actions optimize KV cache placement, replication, retrieval, and lifecycle decisions for subsequent requests. We synthesize existing evidence on KV cache reuse, compression, cross-model mapping, distributed storage, and transfer, and discuss the remaining challenges of integrating these capabilities into one system. Across eight typical workload profiles, our workload-level analytical simulation reports TTFT speedups of 1.25$\times$--13.28$\times$ and input-cost benefits of 1.20$\times$--6.16$\times$.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-09 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2609.06940 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T13:02:21.570Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).