Metraipolis › archive › dc_8fbf173f5294

LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training — arXiv:2608.11919

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.11919
License (as archived)
CC0-1.0
Archived
2026-10-19T05:27:39.181Z
SHA-256 of the archived text
c94f5c2e9b1c330d17b484e4c42cbdcba48ffd0b857a70b2e21c526b1eee004e
Stable URL of this page
https://metraipolis.com/documento/dc_8fbf173f5294

How to cite

LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training — arXiv:2608.11919 — arXiv (en), https://arxiv.org/abs/2608.11919 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8fbf173f5294 (Metraipolis archive dc_8fbf173f5294, sha256 c94f5c2e9b1c330d17b484e4c42cbdcba48ffd0b857a70b2e21c526b1eee004e)
BibTeX
@misc{dc8fbf173f5294,
  title = {LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training — arXiv:2608.11919},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.11919},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8fbf173f5294; sha256 c94f5c2e9b1c330d17b484e4c42cbdcba48ffd0b857a70b2e21c526b1eee004e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.11919. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Training large language models on limited hardware is increasingly a scheduling problem across GPU compute, host memory, PCIe transfer, and storage bandwidth. Existing offloading systems reduce GPU residency, and MegaTrain shows that a CPU-master layer-streaming executor can train large models on a single GPU, but fixed checkpointing and placement heuristics still leave communication exposed on the critical path. We propose LazyTrain, an optimization layer over a layer-streaming executor. LazyTrain formulates checkpoint selection, activation placement, recomputation, and CPU-GPU-NVMe communication overlap as a mixed-integer scheduling problem, then executes the solved policy during training. It further couples 8-bit optimizer states with fast gradient clipping as a single Hybrid 8-bit operator: state compression reduces optimizer-state memory, while fast clipping counteracts the additional CPU-side update overhead. Across H800 experiments from Qwen2.5-3B to Qwen3.6-27B, LazyTrain improves sustained TFLOPS over matched baselines runs by approximately 1.24$\times$; RTX 3090 experiments likewise increase the maximum feasible batch size by one at each model scale. In the primary Qwen3.6-27B H800 MetaMathQA run, LazyTrain reaches 219.95 TFLOPS and 1361 tokens/s at batch size 72, peaks at 68.84\,GB of GPU memory, and obtains 95.42\% exact-match accuracy on the full evaluation split. The source code is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-13 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.11919 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:15:39.223Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).