Metraipolis › archive › dc_4fd18cd78b98

Quantizing Recurrent Language Models Is Easier Than You Think — arXiv:2611.01864

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.01864
License (as archived)
CC0-1.0
Archived
2026-11-03T11:06:47.578Z
SHA-256 of the archived text
930089d2726d39d26a726aa14c0e6b7985747da30ff49ae869b01967b44fd202
Stable URL of this page
https://metraipolis.com/documento/dc_4fd18cd78b98

How to cite

Quantizing Recurrent Language Models Is Easier Than You Think — arXiv:2611.01864 — arXiv (en), https://arxiv.org/abs/2611.01864 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4fd18cd78b98 (Metraipolis archive dc_4fd18cd78b98, sha256 930089d2726d39d26a726aa14c0e6b7985747da30ff49ae869b01967b44fd202)
BibTeX
@misc{dc4fd18cd78b98,
  title = {Quantizing Recurrent Language Models Is Easier Than You Think — arXiv:2611.01864},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01864},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4fd18cd78b98; sha256 930089d2726d39d26a726aa14c0e6b7985747da30ff49ae869b01967b44fd202},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01864. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Linear recurrences are becoming an increasingly common building block in large language models, replacing the growing KV cache of softmax attention with a fixed-size recurrent state that enables efficient decoding. However, the quantization stack has largely been developed around softmax Transformers, whose numerical properties and inference state differ from those of recurrent models. Our goal is to understand these differences and study the design space of quantization methods for recurrent models. In our analysis, we differentiate between two targets. First, extra-state quantization concerns weights and activations outside the recurrent state, whose distributions are shaped indirectly by the sequence mixer during training. In controlled 1.3B parameter models, we find that decayed recurrences develop weaker residual outlier channels than softmax attention and are therefore easier to quantize without incoherence processing. Second, intra-state quantization concerns the recurrent state itself during decode, where repeated updates create a different numerical problem from KV cache quantization. We instantiate our findings as LoReS, a state quantization pipeline combining value-axis scale grouping to match output channel magnitude, decay-based mixed-precision to protect slowly decaying heads or channels, and stochastic rounding to limit error accumulation under repeated requantization. Across Qwen-3.5 9B, Nemotron-H 8B, and Kimi-Linear 48B-A3B, LoReS supports robust 8-bit state quantization and lower effective precision on shorter decode horizons. We implement LoReS in vLLM and demonstrate end-to-end decode speedups.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.01864 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:59:46.054Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).