Metraipolis › archive › dc_7f97d41d7f5b
Just Let Linear States Forget the Distant Past: Prefix Caching via Suffix Replay for Hybrid LLMs — arXiv:2609.33477
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33477
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T20:56:50.396Z
- SHA-256 of the archived text
ab8ef9b4e0f84e36be3adb68a84d89f24bc233480df8ac20a4df73c6d023d67f- Stable URL of this page
https://metraipolis.com/documento/dc_7f97d41d7f5b
How to cite
Just Let Linear States Forget the Distant Past: Prefix Caching via Suffix Replay for Hybrid LLMs — arXiv:2609.33477 — arXiv (en), https://arxiv.org/abs/2609.33477 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7f97d41d7f5b (Metraipolis archive dc_7f97d41d7f5b, sha256 ab8ef9b4e0f84e36be3adb68a84d89f24bc233480df8ac20a4df73c6d023d67f)
BibTeX
@misc{dc7f97d41d7f5b,
title = {Just Let Linear States Forget the Distant Past: Prefix Caching via Suffix Replay for Hybrid LLMs — arXiv:2609.33477},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33477},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7f97d41d7f5b; sha256 ab8ef9b4e0f84e36be3adb68a84d89f24bc233480df8ac20a4df73c6d023d67f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33477. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Hybrid LLMs interleave full-attention layers with linear-attention layers to reduce long-context inference cost, but this structure complicates prefix caching. Full-attention KV caches are token-addressable, whereas linear-attention layers maintain recurrent states that cannot be rolled back to arbitrary prefix boundaries. Existing systems materialize recurrent-state checkpoints, restricting prefix reuse to checkpoint-aligned positions. We present SuffixReplay, the first prefix caching system that lets hybrid LLMs reuse cached prefixes at every cache-supported page boundary without materializing recurrent-state checkpoints. Our key insight is to just let linear states forget the distant past. Modern linear-attention mechanisms use recurrent decay and gating to attenuate the influence of old inputs. Therefore, instead of checkpointing every prefix boundary, SuffixReplay approximates the state at a matched boundary by replaying only a recent suffix of the layer's input hidden states, which we retain as anchors. At the algorithmic level, SuffixReplay combines layer-wise and token-wise anchor sparsity with a bounded replay budget to control storage, computation, and quality. At the system level, it uses an independently managed anchor sidecar and a pipelined replay path to overlap anchor movement and state reconstruction with the native serving pipeline. We evaluate SuffixReplay on three hybrid LLMs: OLMo-Hybrid-7B, Qwen3.5-4B, and Qwen3.6-27B-FP8. Across these models, SuffixReplay retains 91.4-100% of full-prefill quality on average across LongBench and RULER, while using only 0.36-0.51x the amortized per-token storage of SGLang's default 8192-token checkpoint cache. Integrated into SGLang, SuffixReplay reduces median TTFT by 15-70% on branching workloads, sustains 2.3-4.3x SGLang's throughput when the working set exceeds HBM, and matches SGLang on high-hit continuation traffic. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Performance (cs.PF). Pagina del lavoro: https://arxiv.org/abs/2609.33477 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:01.271Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).