Metraipolis › archive › dc_7a8040318151

Layer-wise CPU-GPU Scheduling for LLM Inference on Memory-Limited Consumer GPUs — arXiv:2611.01393

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.01393
License (as archived)
CC0-1.0
Archived
2026-11-03T11:12:38.273Z
SHA-256 of the archived text
6df98fdde734edc579f08deb4bb5f5277d60f48c9580de5059b4895a9df704fb
Stable URL of this page
https://metraipolis.com/documento/dc_7a8040318151

How to cite

Layer-wise CPU-GPU Scheduling for LLM Inference on Memory-Limited Consumer GPUs — arXiv:2611.01393 — arXiv (en), https://arxiv.org/abs/2611.01393 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7a8040318151 (Metraipolis archive dc_7a8040318151, sha256 6df98fdde734edc579f08deb4bb5f5277d60f48c9580de5059b4895a9df704fb)
BibTeX
@misc{dc7a8040318151,
  title = {Layer-wise CPU-GPU Scheduling for LLM Inference on Memory-Limited Consumer GPUs — arXiv:2611.01393},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01393},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7a8040318151; sha256 6df98fdde734edc579f08deb4bb5f5277d60f48c9580de5059b4895a9df704fb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01393. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As large language model inference environments expand toward consumer devices, it remains challenging to efficiently run models that exceed GPU memory capacity in single-GPU settings. The latency of transferring weights that do not fit in GPU memory accounts for about 98\% of inference time and remains a major obstacle. In this paper, we focus on the fact that during the decode phase, layer computation time on the CPU can be shorter than weight transfer time, presenting an opportunity to hide weight transfers by utilizing this computation time. Therefore, we propose a method to hide transfer latency by executing certain layers on the CPU during the decode phase while asynchronously transferring the next layer's weights to the GPU in the background of that computation. To realize this scheduling, we formulate the placement of CPU/GPU computation and weight transfers as a latency minimization problem and perform optimization using an approximation model with relaxed temporal constraints. Evaluation results confirm that in a consumer desktop environment, our approach accelerates OPT-30B inference by 3.5x compared to HuggingFace Accelerate and 6.3x compared to DeepSpeed Zero-Inference.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2611.01393 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T11:00:07.149Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).