Metraipolis › archive › dc_0aa1d6edcdad

The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally — arXiv:2609.01587

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.01587
License (as archived)
CC0-1.0
Archived
2026-10-17T06:41:51.059Z
SHA-256 of the archived text
0ef91ace00f35317ed3f8a4fae4f5457a0a8629b32975f9ac33d921670a3da86
Stable URL of this page
https://metraipolis.com/documento/dc_0aa1d6edcdad

How to cite

The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally — arXiv:2609.01587 — arXiv (en), https://arxiv.org/abs/2609.01587 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0aa1d6edcdad (Metraipolis archive dc_0aa1d6edcdad, sha256 0ef91ace00f35317ed3f8a4fae4f5457a0a8629b32975f9ac33d921670a3da86)
BibTeX
@misc{dc0aa1d6edcdad,
  title = {The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally — arXiv:2609.01587},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.01587},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0aa1d6edcdad; sha256 0ef91ace00f35317ed3f8a4fae4f5457a0a8629b32975f9ac33d921670a3da86},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01587. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Post-training quantization (PTQ) is widely used to reduce the cost of serving large language models (LLMs), but its accuracy cost is uneven and is often tuned per model. We study where quantization damage occurs and how to allocate a small additional precision budget. Using causal mixed-precision intervention as ground truth (raise each layer to 8-bit in turn and measure the accuracy it recovers) across 9 open-weight models in 4 architecture families, we test 3 intuitive hypotheses: that quantization damage lives in task circuits, where the model computes, or in weight statistics. None of them predicts which layers benefit from restored precision. Recovery is instead diffuse: for 8 of 9 models, recovering 75% of the gap takes roughly half the layers; the lone exception, Qwen3-8B, is sharply concentrated. At a matched precision budget, spending it globally on finer quantization granularity beats locally repairing the most recoverable layers for all 8 group-128-compatible models (all but OpenLLaMA, whose width rules out group-128), by 21-52 points, including the concentrated Qwen3-8B. We report 2 secondary findings: the residual is budget-limited (8-bit is near-lossless in our evaluation across RTN, GPTQ, and AWQ), and the location of peak recovery correlates with architecture within a family, though not across families. Within this budget setting, global granularity is a better default than selectively protecting critical layers. More broadly, cheap signals that correlate with quantization damage do not necessarily identify where restoring precision improves accuracy; this must be tested with causal intervention.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.01587 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:33:38.904Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).