Metraipolis › archive › dc_a1cbc632f5bb

A Method for Layer Bit-Width Allocation in LLM Quantization via Performance Maximization Under a Quality-Degradation Constraint — arXiv:2608.28003

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.28003
License (as archived)
CC0-1.0
Archived
2026-10-17T10:48:02.727Z
SHA-256 of the archived text
dd9b348a24b2d66652f8be89878e26e991a0d71b64ab46496daaaccaf3ebae7c
Stable URL of this page
https://metraipolis.com/documento/dc_a1cbc632f5bb

How to cite

A Method for Layer Bit-Width Allocation in LLM Quantization via Performance Maximization Under a Quality-Degradation Constraint — arXiv:2608.28003 — arXiv (en), https://arxiv.org/abs/2608.28003 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a1cbc632f5bb (Metraipolis archive dc_a1cbc632f5bb, sha256 dd9b348a24b2d66652f8be89878e26e991a0d71b64ab46496daaaccaf3ebae7c)
BibTeX
@misc{dca1cbc632f5bb,
  title = {A Method for Layer Bit-Width Allocation in LLM Quantization via Performance Maximization Under a Quality-Degradation Constraint — arXiv:2608.28003},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.28003},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a1cbc632f5bb; sha256 dd9b348a24b2d66652f8be89878e26e991a0d71b64ab46496daaaccaf3ebae7c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.28003. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

This paper proposes a layer bit allocation method for Gemma-3-1B, formulating the problem as performance maximization (latency decrease) given a degradation budget constraint (allowable level of generation quality loss). This approach is different from time- and resource-consuming uniform layer quantization methods that are used in the literature (like GPTQ or AWQ) or allocation methods without proven performance-accelerating effect (like MixLLM or TorchAO). The layer sensitivity profile resulting from our prior work SA-PTQ is applied using the activation pass-through mode inside TensorRT-LLM. For each layer precision is determined individually in blocks, according to a grouping introduced in the prior step (5+5, 10+10, all26), differentiating the contribution of FFN, Attention, and lm_head to the overall speedup. The clock speed was measured for 13 W8A8 variants on an RTX 5090. We find that for FFN and lm_head the time cost of quantization/dequantization is compensated for by the use of integer arithmetic, while for short context lengths, the opposite holds true for Attention: an additional step of quantization slows execution down. We propose a manual implementation of SmoothQuant for TensorRT-LLM which was necessary due to export failures, unavailable for lm_head. The best solution found under joint consideration of all three criteria with minimal degradation was FFN 5+5 with lm_head, providing an 11.0% reduction in latency with negligible quality loss (98.90% Top-1 agreement, +0.85% perplexity degradation). With acceptable quality loss for FFN all26 + lm_head, a speedup up to 19.1% was found possible. We suggest further optimizations: fused attention kernels in INT8, KV-cache quantization, using FP8 instead of INT8 and partial Attention quantization analogous to FFN.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-31 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.28003 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T10:33:42.487Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).