Metraipolis › archive › dc_72b0706dc703
RAQ-MLA: Role-Aware Quantization for Multi-Head Latent Attention Caches — arXiv:2610.33625
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.33625
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-27T07:57:11.506Z
- SHA-256 of the archived text
844115f151f580fe0d744b5d6364807a8745578e73e74c8671adbd76c6cb03ad- Stable URL of this page
https://metraipolis.com/documento/dc_72b0706dc703
How to cite
RAQ-MLA: Role-Aware Quantization for Multi-Head Latent Attention Caches — arXiv:2610.33625 — arXiv (en), https://arxiv.org/abs/2610.33625 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_72b0706dc703 (Metraipolis archive dc_72b0706dc703, sha256 844115f151f580fe0d744b5d6364807a8745578e73e74c8671adbd76c6cb03ad)
BibTeX
@misc{dc72b0706dc703,
title = {RAQ-MLA: Role-Aware Quantization for Multi-Head Latent Attention Caches — arXiv:2610.33625},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.33625},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_72b0706dc703; sha256 844115f151f580fe0d744b5d6364807a8745578e73e74c8671adbd76c6cb03ad},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.33625. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Multi-Head Latent Attention (MLA) has been adopted by recent frontier large language models. Instead of per-head keys and values, MLA caches a compressed KV latent that is up-projected into keys and values at decode time, together with a narrow positional key that carries position information into the attention score. Existing KV-cache quantization methods are designed around the keys and values of standard attention and do not account for what MLA actually caches, which costs accuracy at low bit widths. We analyze the MLA cache both in activation distribution and in how quantization error propagates to the attention output, and show that the two cached tensors play different roles: the latent mainly supplies the values, whereas the positional key dominates the attention score. We therefore propose RAQ-MLA (Role-Aware Quantization for Multi-Head Latent Attention Caches), which fits an invertible transform to each cached tensor against the error that tensor delivers to the attention output: attention-weighted value error for the latent and score perturbation for the positional key. Without training, RAQ-MLA quantizes the full MLA cache to about 2.5 bits per element, a 6.5x increase in theoretical KV-cache capacity, while remaining nearly lossless on reasoning and coding tasks across four MLA models. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.33625 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-27T07:55:30.650Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).