Metraipolis › archive › dc_a89c045d2399
SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference — arXiv:2610.12327
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.12327
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T06:13:52.931Z
- SHA-256 of the archived text
3084c9e66100ee5a8259559c97c3d0c18866aa6d8b7e07f9ec2a893aecd8c48b- Stable URL of this page
https://metraipolis.com/documento/dc_a89c045d2399
How to cite
SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference — arXiv:2610.12327 — arXiv (en), https://arxiv.org/abs/2610.12327 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a89c045d2399 (Metraipolis archive dc_a89c045d2399, sha256 3084c9e66100ee5a8259559c97c3d0c18866aa6d8b7e07f9ec2a893aecd8c48b)
BibTeX
@misc{dca89c045d2399,
title = {SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference — arXiv:2610.12327},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.12327},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a89c045d2399; sha256 3084c9e66100ee5a8259559c97c3d0c18866aa6d8b7e07f9ec2a893aecd8c48b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.12327. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The memory-bound nature of the decoding stage of large language model (LLM) inference incurs significant latency. Layer-wise training-free network pruning approaches guided by the Hessian have been a prominent solution to this problem, as pruning reduces the number of nonzero parameters read from memory during decoding. Nevertheless, typical methods in this line compute the Hessian using pre-collected natural sequences, whereas the model is fed self-generated tokens during decoding, creating a distribution shift between the two sequences. The Hessian calculated on the natural sequence is different from that calculated on the generated sequence. We observe that this discrepancy causes the activation distribution during generation to deviate from that used for pruning, further hurting the pruned model performance. Moreover, most existing LLM pruning methods that bring actual speedup primarily target the sparse matrix-matrix (SpMM) multiplication, providing limited support for the sparse matrix-vector (SpMV) operations, which dominate decoding. To solve these problems, we introduce SparseDecoding, a principled decoding-aware pruning framework tailored for accurate and efficient LLM decoding. Specifically, at the algorithmic axis, SparseDecoding constructs calibration matrices from layer-wise activations collected during the dense-model autoregressive generation, excluding prefill, thereby aligning the pruning objective with the decoding activations. At the system axis, we develop an optimized N:M sparse matrix-vector kernel with bitmask indexing and fixed-step traversal. Substantial empirical results on representative LLMs (Llama-3.1-8B, Llama-3.3-70B, Qwen3-14B / 32B) demonstrate that our method consistently outperforms standard fixed-text calibration on the long-form generation benchmarks while achieving up to 1.48x end-to-end wall-clock decoding speedup on A100 GPUs. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.12327 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T05:47:15.513Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).