Metraipolis › archive › dc_ad26c3cb1ca0
Dispatch Control for Priority-Based, Multi-Tenant Workflows under Multi-Resource External Rate Limits — arXiv:2611.02926
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02926
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:12:44.496Z
- SHA-256 of the archived text
fd27ef50538b347393ca05134369eb49414b307bbc60bcbc8ec0b4c8e5a08381- Stable URL of this page
https://metraipolis.com/documento/dc_ad26c3cb1ca0
How to cite
Dispatch Control for Priority-Based, Multi-Tenant Workflows under Multi-Resource External Rate Limits — arXiv:2611.02926 — arXiv (en), https://arxiv.org/abs/2611.02926 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ad26c3cb1ca0 (Metraipolis archive dc_ad26c3cb1ca0, sha256 fd27ef50538b347393ca05134369eb49414b307bbc60bcbc8ec0b4c8e5a08381)
BibTeX
@misc{dcad26c3cb1ca0,
title = {Dispatch Control for Priority-Based, Multi-Tenant Workflows under Multi-Resource External Rate Limits — arXiv:2611.02926},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02926},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ad26c3cb1ca0; sha256 fd27ef50538b347393ca05134369eb49414b307bbc60bcbc8ec0b4c8e5a08381},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02926. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Workflow engines orchestrate pipelines of activities, which often call external APIs, on a fixed pool of worker slots, the threads that run them. When an API enforces a hard rate limit shared by all tenants and priorities, a throttled call that retries after a backoff keeps its worker slot and starves other tenants. Large Language Model provider APIs are particularly complex: their per-organization caps are low and bind independently on requests and tokens per minute. The operator can neither reconfigure the cap nor observe saturation before a request fails, so classical fair-queuing assumptions do not hold. We present Backpressure, a worker-side admission gate. It checks each call's cost atomically against shared admission state and enforces fairness over requests and tokens. A denied call releases its worker slot and waits on a durable timer, so waiting no longer blocks execution capacity. We compare against a naive retry, the AI gateway LiteLLM, and the built-in throttle of the workflow engine Temporal. On a workload with three priority classes, Backpressure raises fairness (Jain's index over each queue's fraction of its entitled share) from 0.60 to 0.89, where 1.0 is ideal. It also cuts high-priority tail latency roughly 8x. When tokens are the binding limit, all request-based baselines serve every priority equally, whereas Backpressure keeps the priority order and reaches a dominant-resource fairness of ~0.99. Admission costs one round trip to shared state, adds no measurable latency when the cap does not bind, and fails open if shared state is unavailable. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2611.02926 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T11:00:07.164Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).