Metraipolis › archive › dc_ddd0ffc806dd
Who Should Score the Cache? Mid-Depth Layers Are the Best Judges of KV-Cache Eviction — arXiv:2611.02230
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02230
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:24:39.959Z
- SHA-256 of the archived text
61f848d2886213adf4d8ca40a4499c1bf8c115c35a3093593ba7280286227da5- Stable URL of this page
https://metraipolis.com/documento/dc_ddd0ffc806dd
How to cite
Who Should Score the Cache? Mid-Depth Layers Are the Best Judges of KV-Cache Eviction — arXiv:2611.02230 — arXiv (en), https://arxiv.org/abs/2611.02230 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ddd0ffc806dd (Metraipolis archive dc_ddd0ffc806dd, sha256 61f848d2886213adf4d8ca40a4499c1bf8c115c35a3093593ba7280286227da5)
BibTeX
@misc{dcddd0ffc806dd,
title = {Who Should Score the Cache? Mid-Depth Layers Are the Best Judges of KV-Cache Eviction — arXiv:2611.02230},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02230},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ddd0ffc806dd; sha256 61f848d2886213adf4d8ca40a4499c1bf8c115c35a3093593ba7280286227da5},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02230. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Key-value (KV) cache eviction methods rank tokens by attention scores, and each layer scores its own cache. To our knowledge, no eviction method asks which layer should score. We find, across twenty-one models, that the depth of the scoring layer governs eviction quality. On the eighteen models with per-layer caches, spanning 0.5B to 109B, long-context retrieval accuracy collapses at shallow layers, peaks between 0.4 and 0.75 of depth, and degrades at the last layer. The curve arises because a layer judges tokens well only once it has used the question to find the answer, which takes several layers. Modeling eviction as a top-$k$ cut, we compute answer survival probability from prefill attention and reproduce the measured curves. This approach selects the scoring layer using calibration prompts with known answers, and the selected layer is the best layer measured on most models and never significantly below it. It improves on the own-layer default on sixteen of the eighteen models, with significant gains on seven, all at 14B or below, while on larger models the default is already close to the best mid-depth layer. Scoring every cache from the selected layer while keeping one set of tokens for all heads also beats per-head SnapKV by 18 to 43 points on five models up to 14B under tight budgets. This gain carries over to financial filings at 32K context, all at the same memory budget, with no extra forward pass. The same curve explains a penalty in shared-cache models, where several layers read one cache and the layer that writes it scores by default. On Gemma-4-E2B the writer is shallow and loses 28 points to a mid-depth reader at a 10% budget. We also find that in open-book question answering (LongBench) no mid-depth layer beats the own-layer default. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2611.02230 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T18:10:08.045Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).