Metraipolis › archive › dc_33cc67565eba
Compression Technique Interactions in Small Language Models: A Factorial Ablation Study — arXiv:2611.00464
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.00464
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:11:59.424Z
- SHA-256 of the archived text
af31395f3d1f91c4144ea68cdd7fed2590c957b826132b838b8585ba8d56177c- Stable URL of this page
https://metraipolis.com/documento/dc_33cc67565eba
How to cite
Compression Technique Interactions in Small Language Models: A Factorial Ablation Study — arXiv:2611.00464 — arXiv (en), https://arxiv.org/abs/2611.00464 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_33cc67565eba (Metraipolis archive dc_33cc67565eba, sha256 af31395f3d1f91c4144ea68cdd7fed2590c957b826132b838b8585ba8d56177c)
BibTeX
@misc{dc33cc67565eba,
title = {Compression Technique Interactions in Small Language Models: A Factorial Ablation Study — arXiv:2611.00464},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.00464},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_33cc67565eba; sha256 af31395f3d1f91c4144ea68cdd7fed2590c957b826132b838b8585ba8d56177c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00464. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Deploying small language models on resource-constrained hardware requires compression, yet practitioners lack systematic guidance on how quantization, pruning, and post-compression recovery interact when combined. We present a factorial ablation study evaluating 72 compression configurations across four SLM architectures (2.7B-7B parameters), extended with a 48-configuration ordering analysis, benchmark accuracy validation on 30 compressed models, and robustness checks including multi-seed validation and full-corpus perplexity evaluation. INT8 quantization is effectively lossless (<0.5% accuracy degradation), while NF4 quantization combined with L1 magnitude pruning causes model collapse in all 16 tested combinations. LoRA-based fine-tuning reduces degraded perplexity by up to 97% after aggressive pruning; an ablation against standard fine-tuning without a teacher model shows the recovery is driven by LoRA adaptation itself rather than the distillation signal. Compression ordering shows no significant effect among viable configurations (ANOVA p = 0.96) but determines viability for NF4+pruning combinations. INT8 quantization alone provides 40-45% memory reduction with no quality loss. On-device CPU deployment via this http URL confirms these findings transfer off the GPU: 4-bit quantization runs 46-54% faster than 8-bit for a perplexity difference under 0.5 points, while un-structured pruning yields no CPU latency or memory benefit, consistent with its dense storage format. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.00464 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T18:09:53.286Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).