Metraipolis › archive › dc_86b712072b97

NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching — arXiv:2608.22643

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.22643
License (as archived)
CC0-1.0
Archived
2026-10-18T07:50:11.341Z
SHA-256 of the archived text
bfc0ca576371a1652bcfe5b7f73dc71ed62f592834ff0cef3baeab3554af8091
Stable URL of this page
https://metraipolis.com/documento/dc_86b712072b97

How to cite

NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching — arXiv:2608.22643 — arXiv (en), https://arxiv.org/abs/2608.22643 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_86b712072b97 (Metraipolis archive dc_86b712072b97, sha256 bfc0ca576371a1652bcfe5b7f73dc71ed62f592834ff0cef3baeab3554af8091)
BibTeX
@misc{dc86b712072b97,
  title = {NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching — arXiv:2608.22643},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.22643},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_86b712072b97; sha256 bfc0ca576371a1652bcfe5b7f73dc71ed62f592834ff0cef3baeab3554af8091},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.22643. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Deploying large language models on edge devices is increasingly limited by a widening gap between model size and available memory. Existing approaches such as quantization, smaller models, and offloading can raise the effective memory limit, but they still assume that the model can be compressed or partitioned to fit within some budget. We target the harder model-exceeds-memory setting, in which the model remains larger than resident memory throughout execution and storage becomes an active source of weights on the critical path. We observe that MLP activity during autoregressive decoding has strong temporal locality: approximately 82-85% of active neurons persist from one token to the next. This means that most sparse weights needed for the current token are already resident, and only the newly needed rows must be fetched from storage. We present NeuroPrefetcher, a storage-backed LLM inference system that exploits this property through predictive delta prefetching. After layer 0, a single GPU-resident predictor, occupying 2.86% of base model parameters, predicts sparse activity for all downstream MLP layers in one forward pass. The runtime compares these predictions against resident GPU buffers and issues application-scheduled NVMe reads only for incoming delta rows, replacing reactive operating-system demand paging with explicit, model-aware weight movement. On real unified-memory edge hardware, NeuroPrefetcher achieves 7.9-12.0x speedup over this http URL across constrained memory budgets.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-25 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.22643 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T07:47:32.955Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-08-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).