Metraipolis › archive › dc_ff526bef8cc8

Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning — arXiv:2609.06974

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.06974
License (as archived)
CC0-1.0
Archived
2026-10-16T12:45:24.340Z
SHA-256 of the archived text
86800665f65d1306e275bc66f6770781f09b0cd3b124d3606588e9e884bc7a69
Stable URL of this page
https://metraipolis.com/documento/dc_ff526bef8cc8

How to cite

Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning — arXiv:2609.06974 — arXiv (en), https://arxiv.org/abs/2609.06974 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ff526bef8cc8 (Metraipolis archive dc_ff526bef8cc8, sha256 86800665f65d1306e275bc66f6770781f09b0cd3b124d3606588e9e884bc7a69)
BibTeX
@misc{dcff526bef8cc8,
  title = {Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning — arXiv:2609.06974},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.06974},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ff526bef8cc8; sha256 86800665f65d1306e275bc66f6770781f09b0cd3b124d3606588e9e884bc7a69},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.06974. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models achieve strong performance across diverse tasks, but deployment remains costly because of memory, latency, and energy demands. Structured pruning reduces these costs by removing architectural components, yet its recovery stage is often limited by a mismatch between the recovery module's representational capacity and the complexity of the removed knowledge. We call this bottleneck the capacity-knowledge asymmetry and propose OverRep, an Overcomplete Reparameterization framework for structured LLM pruning. Following the principle of "train overcomplete, deploy compact", OverRep temporarily overparameterizes the recovery module during training to absorb complex knowledge distilled from the original model. After recovery, the overcomplete re-parameterization is algebraically merged into a mathematically equivalent compact module, preserving the pruned model's inference-time architecture and computational cost. OverRep further introduces an annealed activation that enables nonlinear training dynamics while converging to a linear regime for exact algebraic merging. Across three backbone families, OverRep improves retained reasoning performance over strong recovery baselines by up to 5.5 and 8.4 points at 25% and 50% pruning, respectively, while keeping memory usage and TFLOPs comparable to existing recovery methods. Our code is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.06974 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T12:30:25.082Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).