Metraipolis › archive › dc_8bbc8cc1a779
Prune Gradually, Adapt Sparsely: Efficient Pruning of Large Language Models at Extreme Sparsity — arXiv:2610.41968
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.41968
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-02T06:59:25.334Z
- SHA-256 of the archived text
7edd6ea1ad95011587f7c64b7f3d76c23643685ebc4a1eb3fa12b647362f73d0- Stable URL of this page
https://metraipolis.com/documento/dc_8bbc8cc1a779
How to cite
Prune Gradually, Adapt Sparsely: Efficient Pruning of Large Language Models at Extreme Sparsity — arXiv:2610.41968 — arXiv (en), https://arxiv.org/abs/2610.41968 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8bbc8cc1a779 (Metraipolis archive dc_8bbc8cc1a779, sha256 7edd6ea1ad95011587f7c64b7f3d76c23643685ebc4a1eb3fa12b647362f73d0)
BibTeX
@misc{dc8bbc8cc1a779,
title = {Prune Gradually, Adapt Sparsely: Efficient Pruning of Large Language Models at Extreme Sparsity — arXiv:2610.41968},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.41968},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8bbc8cc1a779; sha256 7edd6ea1ad95011587f7c64b7f3d76c23643685ebc4a1eb3fa12b647362f73d0},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41968. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Weight pruning can reduce the storage and inference cost of large language models (LLMs), but preserving quality at high sparsity levels remains challenging. Existing methods face a quality-cost trade-off in this regime. One-shot pruning is cheap but degrades sharply as sparsity increases, even when followed by post-pruning recovery. A growing line of work instead optimizes the mask directly from the dense model under the end-to-end objective; among these methods, [Lee et al.,2026], the recent state of the art at extreme sparsity, reaches high quality but at the cost of model-sized optimizer state and auxiliary variables. To address this gap, we propose ETAPS, which prunes gradually and adapts sparsely: it tightens a global sparsity constraint in progressive stages while training a sparse delta with a small, fixed budget. Our key observation is that gradual weight removal keeps each pruning step small enough for sparse adaptation to compensate, allowing the adapted model to provide end-to-end gradients that guide the next pruning decision without model-sized optimizer state. Across six models from 125M to 13B parameters, ETAPS achieves lower perplexity than ELSA at every evaluated sparsity from 50% to 90%, with perplexity reductions of up to 42%. ETAPS is also far cheaper: at 90% sparsity, it prunes LLaMA2-7B on a single H200, where ELSA uses four, with $5.0\times$ smaller total persistent state and 49% fewer GPU-hours. Together, our findings point to a promising direction for achieving high quality in demanding regimes such as extreme sparsity at substantially lower cost. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.41968 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-02T06:56:58.328Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).