Metraipolis › archive › dc_98c0ceaa355b

A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines — arXiv:2611.05991

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.05991
License (as archived)
CC0-1.0
Archived
2026-11-05T18:19:19.704Z
SHA-256 of the archived text
6791dcbf303415f664db6ce9d42ddffe9f2d9c720f026ac28a1160c9d9b876a4
Stable URL of this page
https://metraipolis.com/documento/dc_98c0ceaa355b

How to cite

A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines — arXiv:2611.05991 — arXiv (en), https://arxiv.org/abs/2611.05991 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_98c0ceaa355b (Metraipolis archive dc_98c0ceaa355b, sha256 6791dcbf303415f664db6ce9d42ddffe9f2d9c720f026ac28a1160c9d9b876a4)
BibTeX
@misc{dc98c0ceaa355b,
  title = {A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines — arXiv:2611.05991},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.05991},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_98c0ceaa355b; sha256 6791dcbf303415f664db6ce9d42ddffe9f2d9c720f026ac28a1160c9d9b876a4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05991. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multi-agent LLM applications repeat most of their prompt tokens across requests, and serving engines reuse the cached state of shared prefixes. vLLM and SGLang reuse it at different granularities, and benchmarks of their default deployments report large performance gaps, usually credited to the cache. Such comparisons rarely control for other configuration differences. One of them, whether a long prompt is prefilled in chunks, is set by each engine's defaults from the hardware and the release, and it confounds the comparison. We built a reproducible testbed on Kubernetes and a seeded multi-agent workload generator, and compared the two engines under vendor defaults and with chunked prefill matched. SGLang's configuration is the same in both arms, which makes it an internal control across independently provisioned deployments. Under defaults, vLLM's median time to first token is up to 64x SGLang's. Matching one setting removes 78% of the capacity gap where that gap is largest and 30 to 67% of the latency gap. Near saturation the remaining latency gap reflects queue growth. A fluid queue model driven only by a 4.4% throughput shortfall reproduces it, and shows that such a gap scales with the length of the benchmark run. What survives matching is a tail-latency advantage of 2.4 to 6.4x at equal throughput, consistent with cache-aware ordering of waiting requests. We derive six recommendations for comparing serving engines.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Machine Learning (cs.LG); Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2611.05991 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:13:18.986Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).