Metraipolis › archive › dc_145dd25de371
Repair Changes Which Layers to Remove: Exact Optima for Depth Pruning of Language Models — arXiv:2610.30552
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.30552
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-25T08:37:34.471Z
- SHA-256 of the archived text
36cb0cc8d380e5c8cb8965b245a564cac41ac3a01c55509a6524e0b96e8654cc- Stable URL of this page
https://metraipolis.com/documento/dc_145dd25de371
How to cite
Repair Changes Which Layers to Remove: Exact Optima for Depth Pruning of Language Models — arXiv:2610.30552 — arXiv (en), https://arxiv.org/abs/2610.30552 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_145dd25de371 (Metraipolis archive dc_145dd25de371, sha256 36cb0cc8d380e5c8cb8965b245a564cac41ac3a01c55509a6524e0b96e8654cc)
BibTeX
@misc{dc145dd25de371,
title = {Repair Changes Which Layers to Remove: Exact Optima for Depth Pruning of Language Models — arXiv:2610.30552},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.30552},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_145dd25de371; sha256 36cb0cc8d380e5c8cb8965b245a564cac41ac3a01c55509a6524e0b96e8654cc},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.30552. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Depth pruning removes entire transformer layers to make large language models faster, and recent methods recover quality by replacing the removed layers with a training-free linear map. Which layers to remove, however, is chosen with heuristic scores whose distance from the best choice is unknown. We evaluate every subset of layer groups exactly (about 53,000 pruned models across four families, with and without repair) and find that the best layers to remove depend on the repair: linear repair absorbs the interaction among nearby layers and turns the optimum from a split layout into a contiguous block, so methods that choose layers without the repair miss the repaired optimum by up to 189%. We therefore select layers under the repair that will be applied. A greedy search at the cost of SLEB lowers perplexity by 20% to 66% relative to ReplaceMe on five models up to 27B parameters, and with a task objective it is up to 25 points more accurate given the same data. A second-order expansion of the loss yields a score that selects the repaired block from one forward and one backward pass of the intact model. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2610.30552 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-25T08:31:42.649Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).