Metraipolis › archive › dc_c75f7dfc45f7

Translating KV Caches Between Hybrid-Attention Language Models: MLA Latents, Linear-Attention States and Sparse-Attention Indexers — arXiv:2611.12062

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.12062
License (as archived)
CC0-1.0
Archived
2026-11-10T18:45:10.400Z
SHA-256 of the archived text
a8f4eda4708e50743c53b9c29de411df12df12652a318327a582446031dea26e
Stable URL of this page
https://metraipolis.com/documento/dc_c75f7dfc45f7

How to cite

Translating KV Caches Between Hybrid-Attention Language Models: MLA Latents, Linear-Attention States and Sparse-Attention Indexers — arXiv:2611.12062 — arXiv (en), https://arxiv.org/abs/2611.12062 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c75f7dfc45f7 (Metraipolis archive dc_c75f7dfc45f7, sha256 a8f4eda4708e50743c53b9c29de411df12df12652a318327a582446031dea26e)
BibTeX
@misc{dcc75f7dfc45f7,
  title = {Translating KV Caches Between Hybrid-Attention Language Models: MLA Latents, Linear-Attention States and Sparse-Attention Indexers — arXiv:2611.12062},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.12062},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c75f7dfc45f7; sha256 a8f4eda4708e50743c53b9c29de411df12df12652a318327a582446031dea26e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.12062. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A model that takes over another model's conversation must normally re-read the whole history; a text hand-off shortens the re-read only by dropping content. KV-cache translation instead maps the source's cache into the target's own format, so the target continues from the full context without re-reading it. It works for grouped-query attention (GQA), but open-weights frontier models now cache compressed latents (MLA), sparse-attention indexer keys and fixed-size linear-attention states. We replicate KV-Lingo on a GQA pair and then give, for each of these objects, where to capture and inject it and a mechanism that reaches native quality. We study two hybrid pairs at 16-48B scale in both directions, Kimi-VL-A3B - Kimi-Linear-48B (MLA - MLA + Kimi Delta Attention) and Qwen3.5-27B - Qwen3.8-27B (Gated DeltaNet hybrids), plus a sparse-attention target (Qwen3.8-27B - Qwen3.8-Flash-Next). MLA latents translate with token-wise linear maps; recurrent states are copied with a closed-form correction or synthesised by replaying the target's own recurrence. Exact long-range recall needs token identity: decoding token ids from the source's layer-0 MLA latent raises needle recall from 0.09 to 0.99 at 2-32K and to 0.83-1.00 at 64K, eight times the longest training prompt. The hybrid full-MLA direction is within the paired confidence interval of native quality on every task metric, and same-shape GDN hybrids reach the native level in both directions (apart from 6% refusals inherited from the source), with time to first token 18.7 and 46.8 times shorter than a re-read at 64K tokens. Segment-wise multi-turn transfer stays at the re-prefill level; naive re-translation collapses. On the sparse-attention target, predicted indexer keys recall the needle as well as the target's own keys (one seed). We state as a hypothesis how the recipes transfer to 300B+ hybrid flagships.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.12062 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:37:52.483Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).