Metraipolis › archive › dc_5cbf5f4bfc97
What Breaks and What Comes Back: Capability Damage and Recovery in Depth-Pruned Small Language Models — arXiv:2611.00877
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.00877
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:09:49.358Z
- SHA-256 of the archived text
f0275ecfcc8d55d5ed63a17389cc1e40b5c800e5e357d02c992437b95d1470f5- Stable URL of this page
https://metraipolis.com/documento/dc_5cbf5f4bfc97
How to cite
What Breaks and What Comes Back: Capability Damage and Recovery in Depth-Pruned Small Language Models — arXiv:2611.00877 — arXiv (en), https://arxiv.org/abs/2611.00877 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5cbf5f4bfc97 (Metraipolis archive dc_5cbf5f4bfc97, sha256 f0275ecfcc8d55d5ed63a17389cc1e40b5c800e5e357d02c992437b95d1470f5)
BibTeX
@misc{dc5cbf5f4bfc97,
title = {What Breaks and What Comes Back: Capability Damage and Recovery in Depth-Pruned Small Language Models — arXiv:2611.00877},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.00877},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5cbf5f4bfc97; sha256 f0275ecfcc8d55d5ed63a17389cc1e40b5c800e5e357d02c992437b95d1470f5},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00877. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Depth pruning removes whole Transformer blocks to make a language model smaller and faster, but the damage is not spread evenly across what the model can do, and a short fine-tuning run, called healing, decides what comes back. We measure this for small models used as agents. Four instruction-tuned models of 1.2-1.7B parameters (Llama-3.2-1B, Qwen2.5-1.5B, SmolLM2-1.7B and OLMo-2-1B) are pruned and healed, and six capabilities are tracked: structured output, function calling, instruction following, mathematical reasoning, safety refusal and knowledge. Three results hold across the families. First, function calling and math collapse in every model (retention 0.00-0.07 of the unpruned score), and the order in which capabilities break agrees closely across the four (Kendall's W = 0.92; 0.62 after correcting for chance floors). Second, which parameters are trained matters as much as which data: fully tuning the last three blocks instead of adding low-rank adapters to all blocks costs up to 0.18 in capped mean retention on three families, and one 300k-token pilot per scope detects this. Third, we test a planner that allocates a 1M-token healing budget from 21 short pilot heals. At a fixed budget it raises the weakest capability on two families, although only one of these gains survives a test that also resamples training seeds. Once its 3.48M pilot tokens are counted, uniform healing with the same total number of training tokens matches or beats every planned variant on every family and retains math significantly better on three of them in an item-level test (on one when seeds are also resampled). Healing longer is a strong and simple baseline that planned recovery must beat. Code, per-item predictions and every result are public. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.00877 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T10:59:52.277Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).