Metraipolis › archive › dc_cf7acc45e7fe

MemLLM: End-to-end In-Storage Acceleration of On-device Retrieval-Augmented Generation using Precomputed KV Caches — arXiv:2610.22579

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.22579
License (as archived)
CC0-1.0
Archived
2026-10-19T08:44:20.176Z
SHA-256 of the archived text
050469af83aa156d5b6e3857d33916ce0191c65bd3e053636eadf2af41f374e6
Stable URL of this page
https://metraipolis.com/documento/dc_cf7acc45e7fe

How to cite

MemLLM: End-to-end In-Storage Acceleration of On-device Retrieval-Augmented Generation using Precomputed KV Caches — arXiv:2610.22579 — arXiv (en), https://arxiv.org/abs/2610.22579 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cf7acc45e7fe (Metraipolis archive dc_cf7acc45e7fe, sha256 050469af83aa156d5b6e3857d33916ce0191c65bd3e053636eadf2af41f374e6)
BibTeX
@misc{dccf7acc45e7fe,
  title = {MemLLM: End-to-end In-Storage Acceleration of On-device Retrieval-Augmented Generation using Precomputed KV Caches — arXiv:2610.22579},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.22579},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cf7acc45e7fe; sha256 050469af83aa156d5b6e3857d33916ce0191c65bd3e053636eadf2af41f374e6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22579. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Retrieval-augmented generation (RAG) on edge devices enables personalized large language model (LLM) inference while preserving privacy. However, RAG's inherent length asymmetry makes the prefill phase the dominant bottleneck. While precomputing the KV caches offline alleviates the prefill cost on server systems, the multi-gigabyte caches over the narrow interconnect create a severe I/O wall for on-device systems. To mitigate this, we propose MemLLM, an end-to-end in-storage processing (ISP) architecture that executes the complete attention pipeline based on precomputed KV caches within the SSD. MemLLM utilizes per-channel accelerators to fully utilize the aggregated NAND channel bandwidth and accelerates all KV cache-related computations on the storage-side to minimize PCIe transfers. To efficiently handle the sparse updates required by RAG, MemLLM introduces a fused recomputation-query kernel paired with a hardware-based filter logic to halve the NAND read volume and eliminate write-back penalties. Furthermore, MemLLM's global SRAM-based stateful reduction engine enables a completely DRAM-less dataflow, thereby removing the internal memory bottlenecks and bounding the PCIe egress to an $O(1)$ footprint. A full-stack software stack allows asynchronous, zero-copy transfers from the ISP to the host and provides transparent execution within modern ML frameworks. Evaluated across eight real-world RAG datasets and six on-device LLM models, MemLLM achieves an average end-to-end speedup of 4.06$\times$ over the GPU baseline.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.AR, nuovi lavori del giorno).
Categorie arXiv: Hardware Architecture (cs.AR).
Pagina del lavoro: https://arxiv.org/abs/2610.22579 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:39:54.817Z dalla pagina https://arxiv.org/list/cs.AR/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).