Metraipolis › archive › dc_53fe568ae41a
Activation Sparsity with Weight Approximation for Faster LLM Decoding on Offloaded Weights — arXiv:2610.02598
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.02598
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-05T22:54:44.821Z
- SHA-256 of the archived text
8def646a74201de9163167679dad4996fd63110ec5f82e32ba18cb85a73a514d- Stable URL of this page
https://metraipolis.com/documento/dc_53fe568ae41a
How to cite
Activation Sparsity with Weight Approximation for Faster LLM Decoding on Offloaded Weights — arXiv:2610.02598 — arXiv, https://arxiv.org/abs/2610.02598 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_53fe568ae41a (Metraipolis archive dc_53fe568ae41a, sha256 8def646a74201de9163167679dad4996fd63110ec5f82e32ba18cb85a73a514d)
BibTeX
@misc{dc53fe568ae41a,
title = {Activation Sparsity with Weight Approximation for Faster LLM Decoding on Offloaded Weights — arXiv:2610.02598},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.02598},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_53fe568ae41a; sha256 8def646a74201de9163167679dad4996fd63110ec5f82e32ba18cb85a73a514d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.02598. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Deploying LLMs on consumer-grade GPUs with insufficient memory to hold their weights can result in prohibitively slow inference, because decoding repeatedly transfers offloaded weights from system RAM or flash storage into GPU at much lower bandwidth than local GPU-memory access. Activation sparsity reduces these transfers by skipping weights associated with zero or near-zero activations. However, as more activation contributions are omitted, model quality eventually degrades rapidly, indicating that weights associated with small-magnitude activations collectively influence model quality sharply. In this work, we improve the trade-off between model quality and decoding performance when exploiting activation sparsity. Our key idea is to replace the binary choice of whether or not to read a weight with three options: fully retain it, approximate it using a compressed weight representation, or omit it entirely. SpAx skips weights associated with activations closest to zero, reads approximate weights for smaller-magnitude activations, and reads original weights for the largest-magnitude activations. Smaller-magnitude activations attenuate the errors introduced by approximate weights, while compressed weight representations require fewer bytes to be transferred. With weights offloaded to CPU memory, SpAx speeds up decoding by 3.86X on average (up to 5.57X) with 16-bit weights and 2.06X (up to 2.74X) with 4-bit weights, at a WikiText-2 perplexity increase of at most 10%. With weights offloaded to flash storage, the speedups are 3.31X on average (up to 4.81X) and 1.54X (up to 2.03X). Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.PF, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Performance (cs.PF). Pagina del lavoro: https://arxiv.org/abs/2610.02598 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-05T22:46:55.525Z dalla pagina https://arxiv.org/list/cs.PF/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).