Metraipolis › archive › dc_145dd25de371

Repair Changes Which Layers to Remove: Exact Optima for Depth Pruning of Language Models — arXiv:2610.30552

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.30552
License (as archived)
CC0-1.0
Archived
2026-10-25T08:37:34.471Z
SHA-256 of the archived text
36cb0cc8d380e5c8cb8965b245a564cac41ac3a01c55509a6524e0b96e8654cc
Stable URL of this page
https://metraipolis.com/documento/dc_145dd25de371

How to cite

Repair Changes Which Layers to Remove: Exact Optima for Depth Pruning of Language Models — arXiv:2610.30552 — arXiv (en), https://arxiv.org/abs/2610.30552 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_145dd25de371 (Metraipolis archive dc_145dd25de371, sha256 36cb0cc8d380e5c8cb8965b245a564cac41ac3a01c55509a6524e0b96e8654cc)
BibTeX
@misc{dc145dd25de371,
  title = {Repair Changes Which Layers to Remove: Exact Optima for Depth Pruning of Language Models — arXiv:2610.30552},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.30552},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_145dd25de371; sha256 36cb0cc8d380e5c8cb8965b245a564cac41ac3a01c55509a6524e0b96e8654cc},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.30552. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Depth pruning removes entire transformer layers to make large language models faster, and recent methods recover quality by replacing the removed layers with a training-free linear map. Which layers to remove, however, is chosen with heuristic scores whose distance from the best choice is unknown. We evaluate every subset of layer groups exactly (about 53,000 pruned models across four families, with and without repair) and find that the best layers to remove depend on the repair: linear repair absorbs the interaction among nearby layers and turns the optimum from a split layout into a contiguous block, so methods that choose layers without the repair miss the repaired optimum by up to 189%. We therefore select layers under the repair that will be applied. A greedy search at the cost of SLEB lowers perplexity by 20% to 66% relative to ReplaceMe on five models up to 27B parameters, and with a task objective it is up to 25 points more accurate given the same data. A second-order expansion of the loss yields a score that selects the repaired block from one forward and one backward pass of the intact model.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2610.30552 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-25T08:31:42.649Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).