Metraipolis › archive › dc_4dcbe6692212

VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branch — arXiv:2609.03949

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.03949
License (as archived)
CC0-1.0
Archived
2026-10-17T01:41:56.377Z
SHA-256 of the archived text
e930ad2c317a110e0ad41b7f0ac29dc6734e985c5b47a1f01843a2cd484848b9
Stable URL of this page
https://metraipolis.com/documento/dc_4dcbe6692212

How to cite

VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branch — arXiv:2609.03949 — arXiv (en), https://arxiv.org/abs/2609.03949 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4dcbe6692212 (Metraipolis archive dc_4dcbe6692212, sha256 e930ad2c317a110e0ad41b7f0ac29dc6734e985c5b47a1f01843a2cd484848b9)
BibTeX
@misc{dc4dcbe6692212,
  title = {VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branch — arXiv:2609.03949},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.03949},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4dcbe6692212; sha256 e930ad2c317a110e0ad41b7f0ac29dc6734e985c5b47a1f01843a2cd484848b9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03949. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A long-lived KV cache must be compressed before the queries that will read it exist. Selection by observed attention collapses there: on a NoPE-MLA model, H2O and SnapKV retrieve 0.00 and 0.33 of needles at 8x compression, because a token's importance has not yet been observed. VestigeKV instead derives a sparse attention pattern from a signal the cache already carries, occupying the sparse-attention literature's one unoccupied quadrant: training-free and query-independent. In NoPE-MLA the 64-dimensional decoupled branch is a vestige of RoPE that training repurposes into a salience channel; reading 11% of each row, it partitions the cache into an attended tier and a GPU-resident archive that no row ever leaves, reachable each step by a certified, query-adaptive trigger. Nothing is trained and cache rows are never quantized, so every quality effect attributes to selection and scheduling. On Kimi Linear 48B, retrieval holds at 1.00 under 8x and 0.96 under 32x from 8k to 65k context, with zero gap to full-row selection, and the recall tier holds 128x at 1.00 (8k). Both tiers stay on the GPU, so the win is speed, not memory: the per-step scan reads ~26% of the bytes dense attention would, and on a two-node sglang deployment the crossover sits at ~40k context, reaching 1.18x at 256k and 1.39x at 496k. The mechanism is exclusive to NoPE: the identical operator on a RoPE MLA collapses to 0.08, query-independent salience exists only without rotation, and query-universal exact merging is provably impossible under RoPE. All thresholds were frozen before their data; 20 archived verdicts and 8 closed routes accompany the paper.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.03949 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T01:11:43.618Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).