Metraipolis › archive › dc_1384347ba3ea
Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation — arXiv:2608.24973
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.24973
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T04:15:36.145Z
- SHA-256 of the archived text
aa34cd3d08691dd5353029e5a7518fe2c1656a1d0dd73833973d8f776f4d93cd- Stable URL of this page
https://metraipolis.com/documento/dc_1384347ba3ea
How to cite
Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation — arXiv:2608.24973 — arXiv (en), https://arxiv.org/abs/2608.24973 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1384347ba3ea (Metraipolis archive dc_1384347ba3ea, sha256 aa34cd3d08691dd5353029e5a7518fe2c1656a1d0dd73833973d8f776f4d93cd)
BibTeX
@misc{dc1384347ba3ea,
title = {Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation — arXiv:2608.24973},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.24973},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1384347ba3ea; sha256 aa34cd3d08691dd5353029e5a7518fe2c1656a1d0dd73833973d8f776f4d93cd},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.24973. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
With the rapid development of large-scale pre-trained language models based on Transformer architectures, their high computational and memory costs have become a major obstacle to deployment, especially in resource-constrained environments. Traditional pruning methods typically depend on full gradient-based importance estimation, and they necessitate prior finetuning of the model to achieve satisfactory performance. This process often results in intolerable resource consumption. This paper proposes REP-LIE, a new approach to enable resource-efficient pruning during the process of finetuning. REP-LIE leverages the gradients of LoRA low-rank matrices to estimate the importance of weights without requiring full gradient computation. To address the inherent randomness in importance estimation, a stability score is introduced, serving as the basis for iterative pruning of unimportant model parameters. The pruned model is further finetuned through lightweight updates, eliminating the need for full-parameter optimization in the process of finetuning. Extensive experiments on both medium-scale encoder models and large-scale generative models (LLaMA-7B and Mistral-7B) demonstrate that REP-LIE still achieves competitive performance compared to existing approaches. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-27 in cs.ET). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Emerging Technologies (cs.ET). Pagina del lavoro: https://arxiv.org/abs/2608.24973 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T04:10:43.264Z dalla pagina https://arxiv.org/catchup/cs.ET/2026-08-27?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).