Metraipolis › archive › dc_4252f200d64f

LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction — arXiv:2610.41359

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.41359
License (as archived)
CC0-1.0
Archived
2026-11-03T08:01:27.231Z
SHA-256 of the archived text
a9351a2af0cbb7243bbf5fe41fc15a6b5db2909601d52161168ba3e913940f71
Stable URL of this page
https://metraipolis.com/documento/dc_4252f200d64f

How to cite

LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction — arXiv:2610.41359 — arXiv (en), https://arxiv.org/abs/2610.41359 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4252f200d64f (Metraipolis archive dc_4252f200d64f, sha256 a9351a2af0cbb7243bbf5fe41fc15a6b5db2909601d52161168ba3e913940f71)
BibTeX
@misc{dc4252f200d64f,
  title = {LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction — arXiv:2610.41359},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41359},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4252f200d64f; sha256 a9351a2af0cbb7243bbf5fe41fc15a6b5db2909601d52161168ba3e913940f71},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41359. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Length-aware scheduling reduces request latency in LLM serving by prioritizing shorter requests. However, despite their large model sizes, length predictors can suffer degraded ranking performance under domain and task shifts, and adapting them can require costly retraining. Their prediction overhead can further offset the benefits of scheduling. To improve the generalization and efficiency of length-aware scheduling, we propose \textbf{LEAP}, a length prediction system with fast online adaptation and asynchronous CPU execution. LEAP reuses the serving model's prefill hidden states, which contain information about the output length, and extracts this signal through a linear readout. To adapt to substantial domain and task shifts, it updates the readout at test time through recursive least squares, using completed requests as supervision without gradient computation or historical-data replay. Prediction and updates run asynchronously on CPU, while prediction and initial batch ordering overlap the remaining GPU prefill. Across the evaluated cross-domain transfers, LEAP retains strong ranking performance, and online adaptation improves Spearman correlation by 64.7\% under a larger domain shift. Serving experiments show that LEAP reduces average per-token latency by 15.9--40.8\% compared with the evaluated length predictors, while achieving 1.9\% higher throughput with $30{,}569\times$ fewer predictor parameters than the strongest evaluated baseline.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.41359 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:59:38.711Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).