Metraipolis › archive › dc_de794b2b5486

Repair Changes Which Layers to Remove: Exact Optima for Depth Pruning of Language Models — arXiv:2610.30552

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.30552
License (as archived)
CC0-1.0
Archived
2026-10-23T15:23:21.530Z
SHA-256 of the archived text
abbd10c8d9d596421ba9635a0d24a0ecd426b93dc9f726e7e146a7d2313a8e2e
Stable URL of this page
https://metraipolis.com/documento/dc_de794b2b5486

How to cite

Repair Changes Which Layers to Remove: Exact Optima for Depth Pruning of Language Models — arXiv:2610.30552 — arXiv (en), https://arxiv.org/abs/2610.30552 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_de794b2b5486 (Metraipolis archive dc_de794b2b5486, sha256 abbd10c8d9d596421ba9635a0d24a0ecd426b93dc9f726e7e146a7d2313a8e2e)
BibTeX
@misc{dcde794b2b5486,
  title = {Repair Changes Which Layers to Remove: Exact Optima for Depth Pruning of Language Models — arXiv:2610.30552},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.30552},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_de794b2b5486; sha256 abbd10c8d9d596421ba9635a0d24a0ecd426b93dc9f726e7e146a7d2313a8e2e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.30552. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Depth pruning removes entire transformer layers to make large language models faster, and recent methods recover quality by replacing the removed layers with a training-free linear map. Which layers to remove, however, is chosen with heuristic scores whose distance from the best choice is unknown. We evaluate every subset of layer groups exactly (about 53,000 pruned models across four families, with and without repair) and find that the best layers to remove depend on the repair: linear repair absorbs the interaction among nearby layers and turns the optimum from a split layout into a contiguous block, so methods that choose layers without the repair miss the repaired optimum by up to 189%. We therefore select layers under the repair that will be applied. A greedy search at the cost of SLEB lowers perplexity by 20% to 66% relative to ReplaceMe on five models up to 27B parameters, and with a task objective it is up to 25 points more accurate given the same data. A second-order expansion of the loss yields a score that selects the repaired block from one forward and one backward pass of the intact model.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2610.30552 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-23T15:10:08.338Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).