Metraipolis › archive › dc_1384347ba3ea

Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation — arXiv:2608.24973

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.24973
License (as archived)
CC0-1.0
Archived
2026-10-18T04:15:36.145Z
SHA-256 of the archived text
aa34cd3d08691dd5353029e5a7518fe2c1656a1d0dd73833973d8f776f4d93cd
Stable URL of this page
https://metraipolis.com/documento/dc_1384347ba3ea

How to cite

Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation — arXiv:2608.24973 — arXiv (en), https://arxiv.org/abs/2608.24973 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1384347ba3ea (Metraipolis archive dc_1384347ba3ea, sha256 aa34cd3d08691dd5353029e5a7518fe2c1656a1d0dd73833973d8f776f4d93cd)
BibTeX
@misc{dc1384347ba3ea,
  title = {Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation — arXiv:2608.24973},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.24973},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1384347ba3ea; sha256 aa34cd3d08691dd5353029e5a7518fe2c1656a1d0dd73833973d8f776f4d93cd},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.24973. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

With the rapid development of large-scale pre-trained language models based on Transformer architectures, their high computational and memory costs have become a major obstacle to deployment, especially in resource-constrained environments. Traditional pruning methods typically depend on full gradient-based importance estimation, and they necessitate prior finetuning of the model to achieve satisfactory performance. This process often results in intolerable resource consumption. This paper proposes REP-LIE, a new approach to enable resource-efficient pruning during the process of finetuning. REP-LIE leverages the gradients of LoRA low-rank matrices to estimate the importance of weights without requiring full gradient computation. To address the inherent randomness in importance estimation, a stability score is introduced, serving as the basis for iterative pruning of unimportant model parameters. The pruned model is further finetuned through lightweight updates, eliminating the need for full-parameter optimization in the process of finetuning. Extensive experiments on both medium-scale encoder models and large-scale generative models (LLaMA-7B and Mistral-7B) demonstrate that REP-LIE still achieves competitive performance compared to existing approaches.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-27 in cs.ET).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Emerging Technologies (cs.ET).
Pagina del lavoro: https://arxiv.org/abs/2608.24973 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T04:10:43.264Z dalla pagina https://arxiv.org/catchup/cs.ET/2026-08-27?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).