Metraipolis › archive › dc_b909cfd64cbc

SQuAT: Spherical Lattice Vector Quantization-Aware Training — arXiv:2610.31412

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.31412
License (as archived): CC0-1.0
Archived: 2026-10-23T15:21:24.631Z
SHA-256 of the archived text: 09feddb0e8b0544b2e4fe98f008447b8c5b9ffdac2401e74b4c43a99f43aeb24
Stable URL of this page: https://metraipolis.com/documento/dc_b909cfd64cbc

How to cite

SQuAT: Spherical Lattice Vector Quantization-Aware Training — arXiv:2610.31412 — arXiv (en), https://arxiv.org/abs/2610.31412 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b909cfd64cbc (Metraipolis archive dc_b909cfd64cbc, sha256 09feddb0e8b0544b2e4fe98f008447b8c5b9ffdac2401e74b4c43a99f43aeb24)

BibTeX

@misc{dcb909cfd64cbc,
  title = {SQuAT: Spherical Lattice Vector Quantization-Aware Training — arXiv:2610.31412},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.31412},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b909cfd64cbc; sha256 09feddb0e8b0544b2e4fe98f008447b8c5b9ffdac2401e74b4c43a99f43aeb24},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31412. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Vector quantization (VQ), in particular lattice-based methods, offer strong rate-distortion trade-offs for low-bit large language models (LLMs), yet their behavior under quantization-aware training (VQ-QAT) remains poorly understood. We scale lattice VQ-QAT to billion-parameter LLMs and multi-billion-token training regimes, and introduce SQuAT, a shape-gain VQ-QAT approach trained with a separate straight-through estimator (STE) for the shape and gain. SQuAT decomposes weights into direction and magnitude: normalized directions are quantized using an $E_8$-derived spherical codebook, while gains are quantized independently with low-bit scalar quantizers. We show that, unlike a standard STE applied to the full quantizer, which ignores the shape-gain structure, these separate STEs induce tangent-space updates for directions, yielding an implicit form of spherical (Riemannian) optimization. To enable scalability, we develop a distributed exact nearest-neighbor assignment scheme, in which workers quantize disjoint weight blocks and communicate only discrete indices, reducing VQ-QAT training time by 12.8$\times$ and allowing us to scale it to 10B tokens. Across instruction-tuned LLaMA-3.2-1B/3B, Qwen3-4B, and base Qwen3-1.7B, SQuAT consistently outperforms standard VQ-QAT STE under identical quantizers and initialization, and achieves SOTA VQ-QAT accuracy.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.31412 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-23T15:09:38.399Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).