Metraipolis › archive › dc_55dbbb4d74eb

Semantic Caches for LLM Web Services Leave Their Error Budget Unspent — arXiv:2611.02498

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.02498
License (as archived): CC0-1.0
Archived: 2026-11-03T18:37:24.411Z
SHA-256 of the archived text: 20d84388bfeb52e1a97a122e0cf37bc65d6f3793b0717c3fac834d060cabe7e1
Stable URL of this page: https://metraipolis.com/documento/dc_55dbbb4d74eb

How to cite

Semantic Caches for LLM Web Services Leave Their Error Budget Unspent — arXiv:2611.02498 — arXiv (en), https://arxiv.org/abs/2611.02498 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_55dbbb4d74eb (Metraipolis archive dc_55dbbb4d74eb, sha256 20d84388bfeb52e1a97a122e0cf37bc65d6f3793b0717c3fac834d060cabe7e1)

BibTeX

@misc{dc55dbbb4d74eb,
  title = {Semantic Caches for LLM Web Services Leave Their Error Budget Unspent — arXiv:2611.02498},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02498},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_55dbbb4d74eb; sha256 20d84388bfeb52e1a97a122e0cf37bc65d6f3793b0717c3fac834d060cabe7e1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02498. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Web services built on large language models (LLMs) receive many paraphrases of earlier queries, which a semantic cache answers from stored responses. A reused answer can be wrong, so wrong answers must stay below a share $\delta$ of the queries. Existing caches reuse above a fixed similarity threshold or, like vCache, learn a per-entry rule that bounds each query's error by $\delta$. However, fixed thresholds break the budget on new streams, and per-entry rules need labels on every entry before reusing it. We find an unspent error budget, as vCache realizes only 12-55% of $\delta$ on three public query streams. Surplus spends this budget where reuse is most likely correct. It ranks queries by a risk score shared across paraphrase clusters and opens risk bins as far as an audited error estimate allows. Free labels from every LLM call and a ledger of realized errors keep this threshold safe under drift. On five streams, including ORCAS search queries, Surplus holds $\delta$ in every test cell, which no baseline does. On the shuffled streams it serves 40% more hits than vCache, and its pooled hit rate beats the best baseline holding $\delta$ in each cell by 4.10 points. In a vLLM testbed it cuts the mean latency by 31.59% at the highest load, against 23.67% for vCache.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.02498 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:12:54.576Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).