Metraipolis › archive › dc_ee5aedecc8b5

LoopVQ: Sub 1-bit KV Cache Quantization for Looped Transformers — arXiv:2611.05304

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.05304
License (as archived)
CC0-1.0
Archived
2026-11-04T18:37:37.271Z
SHA-256 of the archived text
036071c225f61e62c15bb45995b92f30d33d3368818685e4c56fd8f90a4fac06
Stable URL of this page
https://metraipolis.com/documento/dc_ee5aedecc8b5

How to cite

LoopVQ: Sub 1-bit KV Cache Quantization for Looped Transformers — arXiv:2611.05304 — arXiv (en), https://arxiv.org/abs/2611.05304 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ee5aedecc8b5 (Metraipolis archive dc_ee5aedecc8b5, sha256 036071c225f61e62c15bb45995b92f30d33d3368818685e4c56fd8f90a4fac06)
BibTeX
@misc{dcee5aedecc8b5,
  title = {LoopVQ: Sub 1-bit KV Cache Quantization for Looped Transformers — arXiv:2611.05304},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.05304},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ee5aedecc8b5; sha256 036071c225f61e62c15bb45995b92f30d33d3368818685e4c56fd8f90a4fac06},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05304. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Looped transformers are a parameter-efficient alternative to standard transformers for language modeling. By reapplying the same layers across multiple loop steps, recent looped language models match the performance of non-looped models several times their size. While this architecture effectively reduces parameter memory, each loop step still stores its own keys and values, so the key-value (KV) cache grows linearly with the number of loop steps and becomes the main memory bottleneck. We observe that the KV cache of looped models is highly redundant along the loop axis and that the loop steps differ substantially in their sensitivity to quantization. Based on these observations, we propose $\textbf{LoopVQ}$, a KV cache vector quantization method for looped transformers that codes channels of multiple loop steps jointly, and fine-tunes the codebooks to match the output of the full-precision model. At 0.5 bits per element, LoopVQ outperforms existing KV cache vector quantization methods for standard LLMs on general, long chain-of-thought reasoning, and long-context retrieval tasks with Ouro and Huginn models. Its design is orthogonal to these methods, and combined with TaSQ, it retains much of the BF16 accuracy even at 0.25 bits per element. We also implement LoopVQ in SGLang, where it nearly halves the decode latency at a 32K-token context and raises the peak serving throughput by up to $4.3\times$ and $6.1\times$ on the 1.4B and 2.6B Ouro models compared to the BF16 baseline.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.05304 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:32:37.493Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).