Metraipolis › archive › dc_db4e529e1cba

Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression — arXiv:2608.24070

Source

arXiv (en)

Original

https://arxiv.org/abs/2608.24070

License (as archived)

CC0-1.0

Archived

2026-10-18T06:22:04.842Z

SHA-256 of the archived text

6202183d268484b4fe782edfd89c51b7f49091b528f25230bddc518f8fa7c90b

Stable URL of this page

https://metraipolis.com/documento/dc_db4e529e1cba

How to cite

Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression — arXiv:2608.24070 — arXiv (en), https://arxiv.org/abs/2608.24070 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_db4e529e1cba (Metraipolis archive dc_db4e529e1cba, sha256 6202183d268484b4fe782edfd89c51b7f49091b528f25230bddc518f8fa7c90b)

@misc{dcdb4e529e1cba,
  title = {Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression — arXiv:2608.24070},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.24070},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_db4e529e1cba; sha256 6202183d268484b4fe782edfd89c51b7f49091b528f25230bddc518f8fa7c90b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.24070. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Prohibitive computational and environmental costs impede the scalable deployment of Large Language Models (LLMs). Traditional compression techniques (sparsity, quantization, low-rank approximations) are typically applied in isolation, and each hits an accuracy-efficiency wall. This thesis proposes the "Compression Trinity," a unified framework that applies the three pillars jointly: sparsity to reduce computation, quantization to minimize memory bandwidth, and low-rank approximations to recover accuracy. To accelerate pretraining, we apply the Trinity to the optimizer and model architecture. MKOR approximates curvature via block-diagonal sparsity and low-rank inversion, maintaining numerical stability for quantized states; it reduces curvature update complexity from $O(d^3)$ to $O(d^2)$ and accelerates convergence by up to 1.85x over KFAC. SLoPe accelerates training by up to 1.25x via a double-pruned backward pass for N:M sparsity, using low-rank "lazy" adapters in the final 1% of training to recover accuracy. For post-training compression, OPTIMA stabilizes static masks in a zero-training regime by formulating weight reconstruction as globally optimal column-wise quadratic programs, improving zero-shot accuracy by up to 3.97%. Given a fine-tuning budget, PATCH breaks the ceiling of static masks by learning a dynamic hybrid sparsity ratio between 0% and 50%, yielding up to 1.38x speedups. Finally, SLiM realizes the full Compression Trinity in one shot, using mathematically derived low-rank adapters to recover information lost to quantization and sparsity, improving accuracy by up to 5.66% over state-of-the-art methods and outperforming uncompressed dense models at equal parameter budgets by 0.6%. Together, these results show that jointly applying the Compression Trinity is essential for efficient, scalable, high-performance LLMs.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-26 in cs.DC).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG); Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2608.24070 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T06:17:57.838Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-08-26?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).