Metraipolis › archive › dc_a1cbc632f5bb
A Method for Layer Bit-Width Allocation in LLM Quantization via Performance Maximization Under a Quality-Degradation Constraint — arXiv:2608.28003
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.28003
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T10:48:02.727Z
- SHA-256 of the archived text
dd9b348a24b2d66652f8be89878e26e991a0d71b64ab46496daaaccaf3ebae7c- Stable URL of this page
https://metraipolis.com/documento/dc_a1cbc632f5bb
How to cite
A Method for Layer Bit-Width Allocation in LLM Quantization via Performance Maximization Under a Quality-Degradation Constraint — arXiv:2608.28003 — arXiv (en), https://arxiv.org/abs/2608.28003 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a1cbc632f5bb (Metraipolis archive dc_a1cbc632f5bb, sha256 dd9b348a24b2d66652f8be89878e26e991a0d71b64ab46496daaaccaf3ebae7c)
BibTeX
@misc{dca1cbc632f5bb,
title = {A Method for Layer Bit-Width Allocation in LLM Quantization via Performance Maximization Under a Quality-Degradation Constraint — arXiv:2608.28003},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.28003},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a1cbc632f5bb; sha256 dd9b348a24b2d66652f8be89878e26e991a0d71b64ab46496daaaccaf3ebae7c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.28003. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
This paper proposes a layer bit allocation method for Gemma-3-1B, formulating the problem as performance maximization (latency decrease) given a degradation budget constraint (allowable level of generation quality loss). This approach is different from time- and resource-consuming uniform layer quantization methods that are used in the literature (like GPTQ or AWQ) or allocation methods without proven performance-accelerating effect (like MixLLM or TorchAO). The layer sensitivity profile resulting from our prior work SA-PTQ is applied using the activation pass-through mode inside TensorRT-LLM. For each layer precision is determined individually in blocks, according to a grouping introduced in the prior step (5+5, 10+10, all26), differentiating the contribution of FFN, Attention, and lm_head to the overall speedup. The clock speed was measured for 13 W8A8 variants on an RTX 5090. We find that for FFN and lm_head the time cost of quantization/dequantization is compensated for by the use of integer arithmetic, while for short context lengths, the opposite holds true for Attention: an additional step of quantization slows execution down. We propose a manual implementation of SmoothQuant for TensorRT-LLM which was necessary due to export failures, unavailable for lm_head. The best solution found under joint consideration of all three criteria with minimal degradation was FFN 5+5 with lm_head, providing an 11.0% reduction in latency with negligible quality loss (98.90% Top-1 agreement, +0.85% perplexity degradation). With acceptable quality loss for FFN all26 + lm_head, a speedup up to 19.1% was found possible. We suggest further optimizations: fused attention kernels in INT8, KV-cache quantization, using FP8 instead of INT8 and partial Attention quantization analogous to FFN. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-31 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.28003 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T10:33:42.487Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).