Metraipolis › archive › dc_009a890c41fe

Trainable Vector Quantization for Large Language Models — arXiv:2610.41822

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.41822
License (as archived)
CC0-1.0
Archived
2026-11-03T08:12:34.971Z
SHA-256 of the archived text
6c7429f9c3f00ef42a9805db95ff75b88a6a2c4bef13116816fceaff8bd1321a
Stable URL of this page
https://metraipolis.com/documento/dc_009a890c41fe

How to cite

Trainable Vector Quantization for Large Language Models — arXiv:2610.41822 — arXiv (en), https://arxiv.org/abs/2610.41822 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_009a890c41fe (Metraipolis archive dc_009a890c41fe, sha256 6c7429f9c3f00ef42a9805db95ff75b88a6a2c4bef13116816fceaff8bd1321a)
BibTeX
@misc{dc009a890c41fe,
  title = {Trainable Vector Quantization for Large Language Models — arXiv:2610.41822},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41822},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_009a890c41fe; sha256 6c7429f9c3f00ef42a9805db95ff75b88a6a2c4bef13116816fceaff8bd1321a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41822. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Vector Quantization (VQ) is a rapidly evolving technique for compressing Large Language Model (LLM) weights, reducing memory usage, and speeding up inference. However, extreme compression deteriorates model performance, which is challenging to recover due to the discrete nature of VQ parameters. Existing methods employ training-free approaches that are limited to layer-wise reconstruction loss minimization. To address this limitation, we propose TVQ, a novel method that leverages straight-through estimation to enable gradient descent-based optimization of vector-quantized LLMs. By reformulating VQ LLM optimization with trainable latent weights, TVQ enables arbitrary objective minimization. We constitute our approach by applying TVQ in a block-wise knowledge distillation paradigm, which balances quality and training efficiency, and propose several practical solutions to accelerate training. In experiments on modern LLMs, we demonstrate that TVQ is compatible with modern VQ schemes. As a result, TVQ achieves state-of-the-art quality in low-bit post-training quantization settings with a modest training budget and straightforward PyTorch implementation. For instance, TVQ achieves near-lossless 2-bit quantization for LLaMA2-70B with only 1% accuracy drop on popular Question-Answering benchmarks.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.41822 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T08:02:38.943Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).