Metraipolis › archive › dc_864bab8aeb9c

SpectrumQuant: Spectrum Alignment for LLM Quantization — arXiv:2610.20303

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.20303
License (as archived)
CC0-1.0
Archived
2026-10-15T14:20:02.101Z
SHA-256 of the archived text
b263e2ffaf43e455b0f0a22739e714f799fc5c4ca04e7979fe1c93e81942a81d
Stable URL of this page
https://metraipolis.com/documento/dc_864bab8aeb9c

How to cite

SpectrumQuant: Spectrum Alignment for LLM Quantization — arXiv:2610.20303 — arXiv (en), https://arxiv.org/abs/2610.20303 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_864bab8aeb9c (Metraipolis archive dc_864bab8aeb9c, sha256 b263e2ffaf43e455b0f0a22739e714f799fc5c4ca04e7979fe1c93e81942a81d)
BibTeX
@misc{dc864bab8aeb9c,
  title = {SpectrumQuant: Spectrum Alignment for LLM Quantization — arXiv:2610.20303},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.20303},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_864bab8aeb9c; sha256 b263e2ffaf43e455b0f0a22739e714f799fc5c4ca04e7979fe1c93e81942a81d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20303. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Orthogonal transforms have emerged as an effective tool for low-bit quantization of large language models (LLMs), as they reshape activation distributions without changing the underlying model function. Existing approaches typically rely either on fixed data-agnostic transforms, such as Hadamard rotations, or on calibration-time optimization of learned rotations. We study rotation selection from first principles and show that maximizing covariance leads to spectral alignment and provides a principled criterion for constructing quantization-friendly orthogonal transforms. Based on this insight, we propose SpectrumQuant, a closed-form rotation method that estimates activation covariance from calibration data and uses it to construct a covariance-aware orthogonal transform for post-training quantization. Our formulation characterizes quantization-friendly rotations as compositions of alignment and mixing components, while recovering Hadamard matrices as a special case. Interestingly, a class of butterfly matrices belongs to this category. In the realm of LLM quantization, SpectrumQuant bridges the gap between data-agnostic structured rotations and learned rotations, providing an effective and efficient data-aware alternative for post-training quantization. Experiments on Llama and Mistral models show that SpectrumQuant is competitive with state-of-the-art rotation strategies while requiring less than one-third of the GPU memory and significantly less time than learned rotations approaches like SpinQuant and DartQuant.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2610.20303 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T13:53:35.727Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).