Metraipolis › archive › dc_abd4086d4748

TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rollouts — arXiv:2608.22788

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.22788
License (as archived)
CC0-1.0
Archived
2026-10-18T07:41:37.206Z
SHA-256 of the archived text
519a7afdfab0d046c2cc3e2c13259678f53f762c54ba02eb3710ec4e03d5f978
Stable URL of this page
https://metraipolis.com/documento/dc_abd4086d4748

How to cite

TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rollouts — arXiv:2608.22788 — arXiv (en), https://arxiv.org/abs/2608.22788 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_abd4086d4748 (Metraipolis archive dc_abd4086d4748, sha256 519a7afdfab0d046c2cc3e2c13259678f53f762c54ba02eb3710ec4e03d5f978)
BibTeX
@misc{dcabd4086d4748,
  title = {TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rollouts — arXiv:2608.22788},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.22788},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_abd4086d4748; sha256 519a7afdfab0d046c2cc3e2c13259678f53f762c54ba02eb3710ec4e03d5f978},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.22788. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large-scale rollouts have become a core component of modern LLM systems, spanning reinforcement learning (RL) post-training, on-policy distillation (OPD), and sampling-heavy evaluation pipelines. Unlike online serving, which is typically optimized for request-level latency and throughput, a small number of long-tail generations can dominate the end-to-end makespan of an entire rollout step. In practice, rollout requests are often routed uniformly across replicas, which can place extremely long generations inside high-concurrency decoding batches. To address this, we present TailSieve, a partial-rollout-guided framework that jointly controls tail routing and replica allocation for LLM rollouts. In an idealized setting with known completion lengths, we show that makespan-optimal routing in the long-tail regime combines tail isolation with load balancing, and that a simple top-k policy closely approximates this offline optimum. Leveraging the observation that long-tail prompts tend to remain long-tailed across policy updates, TailSieve uses partial rollouts as a training-free signal for identifying candidate tail groups. A hierarchical controller then jointly adapts the number of isolated groups and the replica split between the tail and bulk pools using collected response-work history and a measured concurrency-throughput model. TailSieve achieves up to 1.67x routing-only speedup over uniform group routing. The resulting low-concurrency tail pool further enables route-specialized speculative decoding with MTP or DFlash, achieving up to 2.59x speedup over uniform routing. Selected prompts are regenerated under the current policy, preserving on-policy generation and avoiding additional routing-induced length bias in steady state.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-25 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.22788 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T07:37:52.741Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).