Metraipolis › archive › dc_54f757fac99e

Coarse Semantic Blocks for Efficient Training-Free Sparse Attention — arXiv:2611.04313

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04313
License (as archived)
CC0-1.0
Archived
2026-11-04T12:29:51.890Z
SHA-256 of the archived text
0277b3de2a130b5a451afc0b2863150b2c9a24f77ba30fbe9edf2e9ebd8c7534
Stable URL of this page
https://metraipolis.com/documento/dc_54f757fac99e

How to cite

Coarse Semantic Blocks for Efficient Training-Free Sparse Attention — arXiv:2611.04313 — arXiv (en), https://arxiv.org/abs/2611.04313 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_54f757fac99e (Metraipolis archive dc_54f757fac99e, sha256 0277b3de2a130b5a451afc0b2863150b2c9a24f77ba30fbe9edf2e9ebd8c7534)
BibTeX
@misc{dc54f757fac99e,
  title = {Coarse Semantic Blocks for Efficient Training-Free Sparse Attention — arXiv:2611.04313},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04313},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_54f757fac99e; sha256 0277b3de2a130b5a451afc0b2863150b2c9a24f77ba30fbe9edf2e9ebd8c7534},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04313. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We introduce a training-free sparse attention method that builds a prompt-specific key index to accelerate both prefill and decoding. The method clusters the keys into large semantic blocks that group tokens from noncontiguous positions. Each query position selects blocks by scoring their centroids and attends to the original keys and values within them. Queries that select the same block are processed together, so prefill stays fast even though each query chooses its own blocks. Large blocks make centroid scoring and this grouped processing cheap, so each query can attend to more tokens in the same time. Larger blocks with a larger token budget can thus be both faster and more accurate on long-context tasks than smaller blocks with a smaller budget. We evaluate two hybrid LLMs, Qwen3.5-4B and 35B-A3B, on one NVIDIA H200. Including the cost of building and using the index, whole-model wall-clock speedups over dense attention are 2.3-2.4x for prefill and 1.7-2.0x for decoding at 512K tokens, and 2.9-3.0x for prefill at 1M. The 35B-A3B model stays close to dense attention on five RULER tasks through 1M, on LongBench-v2 overall, and on MRCR through 256K. The 4B model loses accuracy mainly on multi-key retrieval, yet outperforms dense Qwen3.5-2B, whose latency is similar at 512K, on all three benchmarks.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.04313 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T12:26:47.269Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).