Metraipolis › archive › dc_38c250a6726b

Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine — arXiv:2611.08684

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.08684
License (as archived)
CC0-1.0
Archived
2026-11-09T22:33:41.154Z
SHA-256 of the archived text
8437d8e335dc97e671d225b498878dfe5cc5b99497b31f1895e19899d2e05336
Stable URL of this page
https://metraipolis.com/documento/dc_38c250a6726b

How to cite

Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine — arXiv:2611.08684 — arXiv (en), https://arxiv.org/abs/2611.08684 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_38c250a6726b (Metraipolis archive dc_38c250a6726b, sha256 8437d8e335dc97e671d225b498878dfe5cc5b99497b31f1895e19899d2e05336)
BibTeX
@misc{dc38c250a6726b,
  title = {Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine — arXiv:2611.08684},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.08684},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_38c250a6726b; sha256 8437d8e335dc97e671d225b498878dfe5cc5b99497b31f1895e19899d2e05336},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.08684. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Clients and proxies in front of an LLM engine retry calls that time out. After a transient overload, these retries can keep the engine from recovering even when calls arrive below its capacity. Since the engine may keep computing for clients that have left, propagating cancellation looks like the natural remedy. In this paper, we test it on vLLM with overload experiments on three open models, and with scheduler traces of one that follow each client attempt to the execution it creates in the engine. We find that with retries on, cancellation alone did not restore service in any run, and only 12-38% of the calls that arrived after the overload were delivered. The traces suggest why: retries that vLLM ran typically started prefill less than half a second before they timed out. Executions that vLLM later aborted took 79-85% of the prompt tokens it computed, and vLLM's prompt-token counter missed a third to half of all computed prompt tokens. A retry budget restored service by shedding retries but left many of the overload's calls undelivered. Request coalescing delivered every call from the overload onward, some late, unless cancellation had already aborted the running execution. Overload controls should thus be judged by the executions that retries add and the calls that clients receive.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.08684 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-09T22:30:47.253Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).