Metraipolis › archive › dc_7920559f7fee
Prefix Keys and Values Matter Late: RIPE Caching for Self-Speculative Decoding in Looped Language Models — arXiv:2610.34436
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.34436
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-27T08:14:23.837Z
- SHA-256 of the archived text
1ef888d0f841ba54317f0be5ce87f29b1fb5d30e046c8f58a2031b9eccf80e6d- Stable URL of this page
https://metraipolis.com/documento/dc_7920559f7fee
How to cite
Prefix Keys and Values Matter Late: RIPE Caching for Self-Speculative Decoding in Looped Language Models — arXiv:2610.34436 — arXiv (en), https://arxiv.org/abs/2610.34436 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7920559f7fee (Metraipolis archive dc_7920559f7fee, sha256 1ef888d0f841ba54317f0be5ce87f29b1fb5d30e046c8f58a2031b9eccf80e6d)
BibTeX
@misc{dc7920559f7fee,
title = {Prefix Keys and Values Matter Late: RIPE Caching for Self-Speculative Decoding in Looped Language Models — arXiv:2610.34436},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.34436},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7920559f7fee; sha256 1ef888d0f841ba54317f0be5ce87f29b1fb5d30e046c8f58a2031b9eccf80e6d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.34436. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
A looped language model runs one block of layers $R$ times per token, so its key-value (KV) cache holds $R$ planes per recurrent layer, and self-speculative decoding (draft with a few loops, verify with all $R$) reads them all. We find that prefix keys and values matter late: at equal perturbation size, loops 25-32 are $e^{15.0}$ (Huginn-0125) and $e^{12.8}$ (Raven-Llama) times more sensitive than loops 2-9, and the planes approach the last one geometrically, at 0.88 and 0.81 per loop. Whether the deepest plane alone suffices then depends on how ripe the late planes are: on Raven, storing only it is indistinguishable from a bf16-to-fp32 change; on Huginn, 1% of positions carry 74% of its error. We turn this into RIPE, which stores the deepest plane everywhere except a window of recent positions kept at full depth. On Huginn, a 64-position window removes 89% of the tail positions at 11.9% of the full cache's memory (1024-token context), and any window at least as long as the draft block makes batched verification block-independent by construction. Nine pre-registered GPU batches back these results. Simple models of acceptance, decoding cost and memory, two of them checked on held-out measurements, account for the measured self-speculative speed-ups ($2.09\times$ with the full cache, $2.58\times$ with the deepest plane) and decompose the $2.24\times$ fixed-memory throughput gain of one shared plane.
Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2610.34436 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:56:15.138Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).