Metraipolis › archive › dc_d649de3fc02b
Transformer-FSQ: Discrete Latents for Extreme KV-Cache Compression — arXiv:2611.09598
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.09598
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-09T22:41:38.265Z
- SHA-256 of the archived text
9a87376e7f49f89936d76b983f3a48481304ef574ea46925f7d938a9ed057a88- Stable URL of this page
https://metraipolis.com/documento/dc_d649de3fc02b
How to cite
Transformer-FSQ: Discrete Latents for Extreme KV-Cache Compression — arXiv:2611.09598 — arXiv (en), https://arxiv.org/abs/2611.09598 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d649de3fc02b (Metraipolis archive dc_d649de3fc02b, sha256 9a87376e7f49f89936d76b983f3a48481304ef574ea46925f7d938a9ed057a88)
BibTeX
@misc{dcd649de3fc02b,
title = {Transformer-FSQ: Discrete Latents for Extreme KV-Cache Compression — arXiv:2611.09598},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.09598},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d649de3fc02b; sha256 9a87376e7f49f89936d76b983f3a48481304ef574ea46925f7d938a9ed057a88},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.09598. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Private, personalised language models must run on local consumer devices, where large, frontier models cannot fit, and retaining long contexts is non-trivial. Sequential memory---the key-value (KV) cache---grows linearly with context length, quickly dwarfing the model itself: for Qwen3-1.7B-Base, 200,000 retained entries require 21 GiB, 6.6 times more than the model weights. Yet the KV-cache compression methods we measured fail beyond moderate compression ratios. We introduce Transformer-FSQ, addressing extreme KV-cache compression by replacing standard key and value projections with a finite scalar quantisation bottleneck. Our cache stores a packed vector of discrete integer codes per token, with no per-token quantisation metadata. Converting a pretrained transformer trains only these projections on 0.25 tokens per original parameter. We evaluate Transformer-FSQ against five baseline families on models from 0.6B to 8B parameters. On long-context retrieval (RULER), from 12$\times$ to 97.5$\times$ Transformer-FSQ is the most accurate method we evaluate, across scales. A fused Triton kernel decodes directly from the integer codes; with baselines given their authors' kernels where available, it decodes at $1.01\times$ (95% interval $0.98$--$1.05$) the throughput of native SDPA attention on the uncompressed Qwen3-8B-Base at $32\times$ and $1.54\times$ at $97.5\times$. Extreme KV-cache compression is therefore a practical way to extend how much context a fixed device can hold. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.09598 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-09T22:33:47.161Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).