Metraipolis › archive › dc_2ff80da0f6ad

Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache — arXiv:2608.30252

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.30252
License (as archived): CC0-1.0
Archived: 2026-10-17T08:37:07.964Z
SHA-256 of the archived text: 0b0cec6014111df1afa27357af56822b527e48666196fa6a8963f6e76589b09e
Stable URL of this page: https://metraipolis.com/documento/dc_2ff80da0f6ad

How to cite

Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache — arXiv:2608.30252 — arXiv (en), https://arxiv.org/abs/2608.30252 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2ff80da0f6ad (Metraipolis archive dc_2ff80da0f6ad, sha256 0b0cec6014111df1afa27357af56822b527e48666196fa6a8963f6e76589b09e)

BibTeX

@misc{dc2ff80da0f6ad,
  title = {Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache — arXiv:2608.30252},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.30252},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2ff80da0f6ad; sha256 0b0cec6014111df1afa27357af56822b527e48666196fa6a8963f6e76589b09e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.30252. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Long-context LLM applications such as document summarization and multi-turn agents require generation from prefixes spanning tens of thousands of tokens, making decoding latency a major bottleneck. Speculative decoding (SD) reduces latency without changing model outputs, but its speedup depends on both accepted draft tokens and draft-step latency: Lightweight drafts are fast but lack the capacity to capture long-range dependencies, whereas strong independent drafts recover acceptance but incur growing KV-access cost at long prefixes. We introduce memory-augmented drafting for long-context SD, equipping a strong independent draft with compressed draft-side KV memory: A lightweight adaptor constructs and incrementally updates this memory to retain distant information and exact recent context. The target verifier retains its full KV cache and applies the standard accept/reject rule, preserving SD's lossless guarantee. Experiments on Llama~3.1-8B and 70B targets at prefix lengths up to 32K show that our method reduces draft-side memory by over 70%. It achieves speedups of up to 2.08x and 3.33x , respectively, over autoregressive decoding.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.30252 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T08:13:26.800Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).