Metraipolis › archive › dc_b09c7dfeb494
LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction — arXiv:2610.41359
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.41359
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T07:51:12.235Z
- SHA-256 of the archived text
802b1161bb78f49f0cdddb47a2f0b2a600d11800f5d4ed71c9718b11bb47120e- Stable URL of this page
https://metraipolis.com/documento/dc_b09c7dfeb494
How to cite
LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction — arXiv:2610.41359 — arXiv (en), https://arxiv.org/abs/2610.41359 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b09c7dfeb494 (Metraipolis archive dc_b09c7dfeb494, sha256 802b1161bb78f49f0cdddb47a2f0b2a600d11800f5d4ed71c9718b11bb47120e)
BibTeX
@misc{dcb09c7dfeb494,
title = {LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction — arXiv:2610.41359},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.41359},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b09c7dfeb494; sha256 802b1161bb78f49f0cdddb47a2f0b2a600d11800f5d4ed71c9718b11bb47120e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41359. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Length-aware scheduling reduces request latency in LLM serving by prioritizing shorter requests. However, despite their large model sizes, length predictors can suffer degraded ranking performance under domain and task shifts, and adapting them can require costly retraining. Their prediction overhead can further offset the benefits of scheduling. To improve the generalization and efficiency of length-aware scheduling, we propose \textbf{LEAP}, a length prediction system with fast online adaptation and asynchronous CPU execution. LEAP reuses the serving model's prefill hidden states, which contain information about the output length, and extracts this signal through a linear readout. To adapt to substantial domain and task shifts, it updates the readout at test time through recursive least squares, using completed requests as supervision without gradient computation or historical-data replay. Prediction and updates run asynchronously on CPU, while prediction and initial batch ordering overlap the remaining GPU prefill. Across the evaluated cross-domain transfers, LEAP retains strong ranking performance, and online adaptation improves Spearman correlation by 64.7\% under a larger domain shift. Serving experiments show that LEAP reduces average per-token latency by 15.9--40.8\% compared with the evaluated length predictors, while achieving 1.9\% higher throughput with $30{,}569\times$ fewer predictor parameters than the strongest evaluated baseline.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.41359 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:49:31.055Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).