Metraipolis › archive › dc_c85fcc3e2dc3

Inference First, Training Underneath: Layer-Granular LoRA Co-Serving with LayerLoom — arXiv:2611.00652

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.00652
License (as archived)
CC0-1.0
Archived
2026-11-03T11:20:42.692Z
SHA-256 of the archived text
b2de764548ecd60b4a8e30922847c5dde6cd87f39e7d0bb0896e2547dbb19f99
Stable URL of this page
https://metraipolis.com/documento/dc_c85fcc3e2dc3

How to cite

Inference First, Training Underneath: Layer-Granular LoRA Co-Serving with LayerLoom — arXiv:2611.00652 — arXiv (en), https://arxiv.org/abs/2611.00652 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c85fcc3e2dc3 (Metraipolis archive dc_c85fcc3e2dc3, sha256 b2de764548ecd60b4a8e30922847c5dde6cd87f39e7d0bb0896e2547dbb19f99)
BibTeX
@misc{dcc85fcc3e2dc3,
  title = {Inference First, Training Underneath: Layer-Granular LoRA Co-Serving with LayerLoom — arXiv:2611.00652},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.00652},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c85fcc3e2dc3; sha256 b2de764548ecd60b4a8e30922847c5dde6cd87f39e7d0bb0896e2547dbb19f99},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00652. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Serving and fine-tuning a large language model on the same GPUs can turn the idle capacity between inference bursts into training progress, and low-rank adaptation (LoRA) makes this co-location cheap because both workloads read a single frozen copy of the base weights. The difficulty lies in the fine-tuning backward pass, which has no counterpart in inference. Current co-serving systems schedule backward work per serving iteration: each iteration carries a fixed backward budget, and requests that arrive in the meantime wait for it to finish, so fine-tuning dictates inference latency whenever load spikes. We observe that the backward pass of a LoRA microbatch decomposes into per-layer tasks over the same weights that the inference forward pass visits, which allows backward work to be scheduled at layer rather than iteration granularity. Building on this observation, LayerLoom attaches same-layer backward tasks to the inference forward pass and, at each layer, runs only as many of them as the latency slack of in-flight requests permits. To keep a ready task at every layer, LayerLoom maintains a wavefront of fine-tuning microbatches staggered across layers, and it bounds the wavefront's activation memory by offloading microbatch state to host memory and prefetching it ahead of the layer where the scheduler plans to use it. A fused kernel serves the paired forward and backward GEMMs from a single read of each base weight. Because all backward work of an iteration finishes within its forward pass, new requests join the next iteration as in continuous batching. On an RTX 4090 GPU running Llama-3.1-8B under a bursty production trace, LayerLoom raises joint SLO attainment by 13.5-21.8 points over prior works with fine-tuning throughput that is comparable or up to 12% higher, and it reaches the same training losses.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.00652 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T11:02:37.462Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).