Metraipolis › archive › dc_c33abfa8b729
A Layer Importance Metric for Quantization Accounting for the Speed-Quality Trade-off in Autoregressive Models — arXiv:2608.26926
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.26926
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T16:12:23.972Z
- SHA-256 of the archived text
5e7c522d75744a930ed99e7028ba1c7ad3129d0ade3ef4d2509393c70f482d61- Stable URL of this page
https://metraipolis.com/documento/dc_c33abfa8b729
How to cite
A Layer Importance Metric for Quantization Accounting for the Speed-Quality Trade-off in Autoregressive Models — arXiv:2608.26926 — arXiv (en), https://arxiv.org/abs/2608.26926 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c33abfa8b729 (Metraipolis archive dc_c33abfa8b729, sha256 5e7c522d75744a930ed99e7028ba1c7ad3129d0ade3ef4d2509393c70f482d61)
BibTeX
@misc{dcc33abfa8b729,
title = {A Layer Importance Metric for Quantization Accounting for the Speed-Quality Trade-off in Autoregressive Models — arXiv:2608.26926},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.26926},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c33abfa8b729; sha256 5e7c522d75744a930ed99e7028ba1c7ad3129d0ade3ef4d2509393c70f482d61},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26926. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Small language models (sLLMs) are nowadays hosted on devices with limited memory and computational budget. In an autoregressive setup, inference is memory-bandwidth bound: uniform quantization is often detrimental to such models, since their architecture has limited redundancies and only a few layers are not very sensitive to lower precision. We propose a composite metric that combines two orthogonal criteria: information retention (measured in terms of a normalized SQNR-based coefficient) and throughput gains (modeled using a roofline-based latency analysis). By profiling Gemma 3 1B, we find that Feed-Forward Network blocks and the embedding matrix are the most promising targets for acceleration. For each candidate, we estimate a normalized quality score based on simulated quantization and a normalized speed score based on roofline modeling with no actual execution needed. We combine the two scores in a composite priority coefficient, allowing us to tune the trade-off between speed and quality as needed. Our metric is general and can be used to prioritize individual blocks, their projection sublayers, or transformer layers as a whole. We evaluate our approach on several model architectures, showing that our estimates have at around 4% prediction error for the accelerated speedup. We find that our method generally allocates more resources to the most expressive layers compared to evolutionary search, specialized accelerators, or Shapley-value-based approaches that require expensive approximate inference. Our analytical approach makes sLLM quantization a predictable engineering task. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.26926 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T15:34:49.933Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).