Metraipolis › archive › dc_46a75075c19d

COEC: Calibrated Orthogonal-Equivalence Compensation for Structured Pruning of Large Language Models — arXiv:2608.21142

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.21142
License (as archived): CC0-1.0
Archived: 2026-10-18T10:31:12.384Z
SHA-256 of the archived text: 2c0141d1380869759dfdbcec9d7f7bd25ba8a04ee6bbfd2a10d43f888f780b76
Stable URL of this page: https://metraipolis.com/documento/dc_46a75075c19d

How to cite

COEC: Calibrated Orthogonal-Equivalence Compensation for Structured Pruning of Large Language Models — arXiv:2608.21142 — arXiv (en), https://arxiv.org/abs/2608.21142 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_46a75075c19d (Metraipolis archive dc_46a75075c19d, sha256 2c0141d1380869759dfdbcec9d7f7bd25ba8a04ee6bbfd2a10d43f888f780b76)

BibTeX

@misc{dc46a75075c19d,
  title = {COEC: Calibrated Orthogonal-Equivalence Compensation for Structured Pruning of Large Language Models — arXiv:2608.21142},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.21142},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_46a75075c19d; sha256 2c0141d1380869759dfdbcec9d7f7bd25ba8a04ee6bbfd2a10d43f888f780b76},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.21142. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Structured pruning reduces the size and inference cost of large language models (LLMs) by removing weight columns, but the resulting output error can degrade accuracy. Existing training-free compensation methods use an additive bias or a single orthogonal rotation on the output side of the retained weight. These corrections leave its input singular frame unchanged and therefore limit how the retained weight can adapt after column removal. We propose COEC (Calibrated Orthogonal-Equivalence Compensation), a training-free compensation framework that applies alternating left and right orthogonal rotations to the retained weight. The right rotation is optimized on a reduced Stiefel manifold, while singular values are rescaled using generalized cross-validation to select the regularization strength for each layer. COEC further tempers the calibration Gram matrix to reduce the dominance of high-energy activation directions and introduces an alignment penalty that preserves the geometric relation between adjacent attention this http URL components use second-order statistics from a small calibration set and require neither backpropagation through the LLM nor retraining of the model parameters. COEC is independent of the column pruning criterion and can be applied to multiple structured pruning methods. Experiments on the Llama-3, Llama-3.1, and Qwen2.5 model families across multiple structured sparsity levels show that COEC improves perplexity on every model and zero-shot accuracy in most settings over existing compensation methods, with larger gains at higher sparsity. These results show that post-pruning compensation can recover part of the performance lost to column removal.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-24 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.21142 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T10:27:55.954Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).