Metraipolis › archive › dc_84d82efcf8e1
Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management — arXiv:2609.06940
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.06940
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T13:07:13.758Z
- SHA-256 of the archived text
d773d8275346561cfe054b3c760043fa9f89d27540d9f999012cfb8d1b222d71- Stable URL of this page
https://metraipolis.com/documento/dc_84d82efcf8e1
How to cite
Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management — arXiv:2609.06940 — arXiv (en), https://arxiv.org/abs/2609.06940 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_84d82efcf8e1 (Metraipolis archive dc_84d82efcf8e1, sha256 d773d8275346561cfe054b3c760043fa9f89d27540d9f999012cfb8d1b222d71)
BibTeX
@misc{dc84d82efcf8e1,
title = {Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management — arXiv:2609.06940},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.06940},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_84d82efcf8e1; sha256 d773d8275346561cfe054b3c760043fa9f89d27540d9f999012cfb8d1b222d71},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.06940. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language model (LLM) inference increasingly spans models that differ in size, capability, price, and provider. This shift creates two costs for developers. One is the integration cost of choosing among and switching between many models. The other is the inference cost of rebuilding a KV cache when it is unavailable or incompatible with the selected model. We define and analyze the Unified AI Gateway as a system setting for an edge-deployed AI traffic hub. It coordinates model routing, KV cache management, and compute placement across end devices, edge resources, and cloud model services. At request time, the gateway jointly selects a target model, an execution site, and a KV cache action under task-quality, latency, cost, and resource constraints. In parallel, background cache-management actions optimize KV cache placement, replication, retrieval, and lifecycle decisions for subsequent requests. We synthesize existing evidence on KV cache reuse, compression, cross-model mapping, distributed storage, and transfer, and discuss the remaining challenges of integrating these capabilities into one system. Across eight typical workload profiles, our workload-level analytical simulation reports TTFT speedups of 1.25$\times$--13.28$\times$ and input-cost benefits of 1.20$\times$--6.16$\times$. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-09 in cs.DC). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2609.06940 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T13:02:21.570Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).