Metraipolis › archive › dc_33cc67565eba

Compression Technique Interactions in Small Language Models: A Factorial Ablation Study — arXiv:2611.00464

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.00464
License (as archived)
CC0-1.0
Archived
2026-11-03T18:11:59.424Z
SHA-256 of the archived text
af31395f3d1f91c4144ea68cdd7fed2590c957b826132b838b8585ba8d56177c
Stable URL of this page
https://metraipolis.com/documento/dc_33cc67565eba

How to cite

Compression Technique Interactions in Small Language Models: A Factorial Ablation Study — arXiv:2611.00464 — arXiv (en), https://arxiv.org/abs/2611.00464 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_33cc67565eba (Metraipolis archive dc_33cc67565eba, sha256 af31395f3d1f91c4144ea68cdd7fed2590c957b826132b838b8585ba8d56177c)
BibTeX
@misc{dc33cc67565eba,
  title = {Compression Technique Interactions in Small Language Models: A Factorial Ablation Study — arXiv:2611.00464},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.00464},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_33cc67565eba; sha256 af31395f3d1f91c4144ea68cdd7fed2590c957b826132b838b8585ba8d56177c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00464. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Deploying small language models on resource-constrained hardware requires compression, yet practitioners lack systematic guidance on how quantization, pruning, and post-compression recovery interact when combined. We present a factorial ablation study evaluating 72 compression configurations across four SLM architectures (2.7B-7B parameters), extended with a 48-configuration ordering analysis, benchmark accuracy validation on 30 compressed models, and robustness checks including multi-seed validation and full-corpus perplexity evaluation. INT8 quantization is effectively lossless (<0.5% accuracy degradation), while NF4 quantization combined with L1 magnitude pruning causes model collapse in all 16 tested combinations. LoRA-based fine-tuning reduces degraded perplexity by up to 97% after aggressive pruning; an ablation against standard fine-tuning without a teacher model shows the recovery is driven by LoRA adaptation itself rather than the distillation signal. Compression ordering shows no significant effect among viable configurations (ANOVA p = 0.96) but determines viability for NF4+pruning combinations. INT8 quantization alone provides 40-45% memory reduction with no quality loss. On-device CPU deployment via this http URL confirms these findings transfer off the GPU: 4-bit quantization runs 46-54% faster than 8-bit for a perplexity difference under 0.5 points, while un-structured pruning yields no CPU latency or memory benefit, consistent with its dense storage format.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.00464 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:09:53.286Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).