Metraipolis › archive › dc_fbbe413662db

ShamAN-Q: Shampoo Augmented NanoQuant for Sub-1-bit LLM Weights — arXiv:2609.38521

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.38521
License (as archived): CC0-1.0
Archived: 2026-10-07T12:43:51.507Z
SHA-256 of the archived text: 259624fab1848c94571b08b31de3c838cbab92423f7dac722129299173740a66
Stable URL of this page: https://metraipolis.com/documento/dc_fbbe413662db

How to cite

ShamAN-Q: Shampoo Augmented NanoQuant for Sub-1-bit LLM Weights — arXiv:2609.38521 — arXiv (en), https://arxiv.org/abs/2609.38521 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fbbe413662db (Metraipolis archive dc_fbbe413662db, sha256 259624fab1848c94571b08b31de3c838cbab92423f7dac722129299173740a66)

BibTeX

@misc{dcfbbe413662db,
  title = {ShamAN-Q: Shampoo Augmented NanoQuant for Sub-1-bit LLM Weights — arXiv:2609.38521},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.38521},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fbbe413662db; sha256 259624fab1848c94571b08b31de3c838cbab92423f7dac722129299173740a66},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38521. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We introduce ShamAN-Q, a sub-1-bit post-training quantization method that extends NanoQuant by replacing each its diagonal reconstruction geometry with a tractable dense curvature metric, using a general paradigm popularized by the Shampoo optimizer. For each linear weight, ShamAN-Q fits a Kronecker product to the empirical Fisher information matrix of a small calibration set by Kullback--Leibler minimization, forming a Mahalanobis reconstruction loss from the result. The continuous ADMM updates from NanoQuant become solutions to Sylvester equations, while its discrete projection and deployment format remain unchanged. Because the curvature is local to a given set of weights, ShamAN-Q re-measures the input curvature statistic for each layer immediately before layer factorization, periodically refreshing all statistics on the partially quantized model. ShamAN-Q also redistributes the uniform rank from NanoQuant across layers at the same total number of bits. On Qwen3-Base, ShamAN-Q lowers WikiText-2 perplexity at $\approx$1 bpw from 27.56 to 22.96 (0.6B), 19.21 to 16.72 (1.7B), and 14.29 to 13.80 (4B) while matching or improving zero-shot accuracy on the Eleuther LM Evaluation Harness. On 0.6B, ShamAN-Q at $\approx$0.8 bpw matches the published perplexity of NanoQuant at $\approx$1.0 bpw.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2609.38521 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:13.893Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).