Metraipolis › archive › dc_38e2af9e3dff

The Devil Is in the Reconstruction Loss Scale: Rethinking Optimization in LLM Quantization — arXiv:2610.00983

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.00983
License (as archived)
CC0-1.0
Archived
2026-10-06T12:12:19.976Z
SHA-256 of the archived text
c1eff9b03c9b0b11c8501b7d4907f0e511da932b8e44625d9cdf01fece2952ce
Stable URL of this page
https://metraipolis.com/documento/dc_38e2af9e3dff

How to cite

The Devil Is in the Reconstruction Loss Scale: Rethinking Optimization in LLM Quantization — arXiv:2610.00983 — arXiv (en), https://arxiv.org/abs/2610.00983 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_38e2af9e3dff (Metraipolis archive dc_38e2af9e3dff, sha256 c1eff9b03c9b0b11c8501b7d4907f0e511da932b8e44625d9cdf01fece2952ce)
BibTeX
@misc{dc38e2af9e3dff,
  title = {The Devil Is in the Reconstruction Loss Scale: Rethinking Optimization in LLM Quantization — arXiv:2610.00983},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.00983},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_38e2af9e3dff; sha256 c1eff9b03c9b0b11c8501b7d4907f0e511da932b8e44625d9cdf01fece2952ce},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00983. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Post-training quantization (PTQ) methods typically use sequential quantization that partitions a pre-trained LLM into a series of units (e.g., transformer blocks), with one unit quantized at each stage. State-of-the-art PTQ methods are predominantly learning-based, optimizing auxiliary quantization parameters (e.g., scaling factors, rotation matrices, clipping thresholds, and adapters) via gradient descent to minimize a reconstruction loss. A common practice is to use mean squared error (MSE) as the reconstruction loss function, yet its induced optimization behavior remains largely unexplored. In this work, we take a holistic view of sequential quantization and systematically investigate how optimization evolves from the first quantization stage to the last, aiming for a deep understanding of optimization in learning-based PTQ schemes. Through extensive empirical studies spanning representative learning-based PTQ methods, LLM families, model scales, architectures, quantization settings and various tasks, we consistently uncover Optimization Imbalance: reconstruction loss magnitudes vary dramatically across stages, accompanied by highly uneven gradient magnitudes and parameter updates under MSE. We term the cross-stage range of loss magnitudes the reconstruction loss scale, and reveal that MSE translates the unexpectedly large reconstruction loss scale into highly uneven gradient magnitudes, which in turn lead to uneven optimization strength across quantization stages. This finding suggests a general principle for improving learning-based PTQ: optimization strength across stages should be decoupled from the reconstruction loss scale. Theoretically, we show that root mean squared error (RMSE) variants defined at the sample, channel, token, and element levels naturally realize this principle through implicit gradient normalization, outperforming MSE significantly as a drop-in replacement.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.00983 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:28.142Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).