Metraipolis › archive › dc_a41d491fd589

What Matters for Aggressive Decoding-Time KV Eviction? Temporal Aggregation and Ranking Preservation — arXiv:2609.03515

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.03515
License (as archived)
CC0-1.0
Archived
2026-10-17T01:10:49.528Z
SHA-256 of the archived text
01c10c4dda820cd081980b18b266341d913f69817af2b8110ccea5507edd0adc
Stable URL of this page
https://metraipolis.com/documento/dc_a41d491fd589

How to cite

What Matters for Aggressive Decoding-Time KV Eviction? Temporal Aggregation and Ranking Preservation — arXiv:2609.03515 — arXiv (en), https://arxiv.org/abs/2609.03515 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a41d491fd589 (Metraipolis archive dc_a41d491fd589, sha256 01c10c4dda820cd081980b18b266341d913f69817af2b8110ccea5507edd0adc)
BibTeX
@misc{dca41d491fd589,
  title = {What Matters for Aggressive Decoding-Time KV Eviction? Temporal Aggregation and Ranking Preservation — arXiv:2609.03515},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.03515},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a41d491fd589; sha256 01c10c4dda820cd081980b18b266341d913f69817af2b8110ccea5507edd0adc},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03515. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Decoding-time KV cache compression research focuses heavily on designing better token scoring functions, while the temporal rule that aggregates scores across decode steps is often treated as an implementation detail. Under aggressive KV compression, we find that exponential-moving-average (EMA) aggregation makes approximately order-preserving scorer modifications largely indistinguishable at the eviction-set level. Value-norm and entropy variants remain highly correlated with attention and produce nearly unchanged retention sets, whereas KeyDiff, key norm, recency, and a learned scorer alter the ranking and degrade substantially. We associate this stability with the evaluated aggregation, which couples layer weighting and temporal retention. Building on this observation, we introduce InertiaKV, an EMA-based decoding-time eviction method, and InertiaKV-Lazy, its periodic-refresh variant, which yields 1.34-1.46x decode throughput relative to full refresh InertiaKV. We also study Score-Free decoding as a separate empirical operating point: it scores the full context once at the first decode step, freezes that ranking, and incurs an average quality change of +0.03 while removing all subsequent scoring. Across six open-weight backbones and the LongBench, LongBench-v2, and RULER benchmarks, the results identify temporal aggregation and ranking preservation as distinct, consequential design factors; they do not imply that scoring quality is irrelevant in general.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.03515 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T00:28:02.855Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).