Metraipolis › archive › dc_72b10ff66924
HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing — arXiv:2609.26368
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.26368
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T14:36:44.820Z
- SHA-256 of the archived text
453bdf94162e63d7c7a8cc6d76763ed322d70afd5711e5a6012a017009927d5a- Stable URL of this page
https://metraipolis.com/documento/dc_72b10ff66924
How to cite
HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing — arXiv:2609.26368 — arXiv (en), https://arxiv.org/abs/2609.26368 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_72b10ff66924 (Metraipolis archive dc_72b10ff66924, sha256 453bdf94162e63d7c7a8cc6d76763ed322d70afd5711e5a6012a017009927d5a)
BibTeX
@misc{dc72b10ff66924,
title = {HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing — arXiv:2609.26368},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.26368},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_72b10ff66924; sha256 453bdf94162e63d7c7a8cc6d76763ed322d70afd5711e5a6012a017009927d5a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.26368. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Long-horizon and multi-turn agents typically generate short actions and process long observations from tools and environments. This growing context demands efficient prefill, compact KV-cache storage, and accurate long-context retrieval. To meet these demands, we introduce HySparse2, a hybrid sparse attention architecture with two-level KV sharing. At the outer level, KV Bridging adopts a YOCO-style self-decoder and cross-decoder structure, but bridges only full-attention layers. The self-decoder uses hybrid sliding-window attention (SWA), while the cross-decoder uses hybrid sparse attention. The KV caches for full-attention layers in the cross-decoder are generated from the hidden states of full-attention layers in the self-decoder. At the inner level, HySparse2 retains HySparse's core KV Reuse design with two refinements. First, it replaces block-level sparsity with token-level sparsity for finer long-context retrieval. Second, it removes the separate SWA branch from sparse layers and instead forces a sliding window of recent tokens into the sparse selection. This two-level KV sharing allows all cross-decoder KV caches to be constructed from self-decoder hidden states. Prefill can therefore exit after the self-decoder, skipping all cross-decoder layers. On an 80B-A3B MoE model, HySparse2 outperforms HySparse and Hybrid SWA on long-context retrieval and multi-turn agentic tasks, while substantially reducing prefill computation and KV-cache storage. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-23 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.26368 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T13:46:15.742Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).