Metraipolis › archive › dc_befb40ba29f4

Coarse Semantic Blocks for Efficient Training-Free Sparse Attention — arXiv:2611.04313

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04313
License (as archived)
CC0-1.0
Archived
2026-11-04T18:36:37.028Z
SHA-256 of the archived text
e54bcd2bab166e011e5c9bade83891c3b9e17c309e56bbb14f3e63b753f033de
Stable URL of this page
https://metraipolis.com/documento/dc_befb40ba29f4

How to cite

Coarse Semantic Blocks for Efficient Training-Free Sparse Attention — arXiv:2611.04313 — arXiv (en), https://arxiv.org/abs/2611.04313 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_befb40ba29f4 (Metraipolis archive dc_befb40ba29f4, sha256 e54bcd2bab166e011e5c9bade83891c3b9e17c309e56bbb14f3e63b753f033de)
BibTeX
@misc{dcbefb40ba29f4,
  title = {Coarse Semantic Blocks for Efficient Training-Free Sparse Attention — arXiv:2611.04313},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04313},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_befb40ba29f4; sha256 e54bcd2bab166e011e5c9bade83891c3b9e17c309e56bbb14f3e63b753f033de},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04313. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We introduce a training-free sparse attention method that builds a prompt-specific key index to accelerate both prefill and decoding. The method clusters the keys into large semantic blocks that group tokens from noncontiguous positions. Each query position selects blocks by scoring their centroids and attends to the original keys and values within them. Queries that select the same block are processed together, so prefill stays fast even though each query chooses its own blocks. Large blocks make centroid scoring and this grouped processing cheap, so each query can attend to more tokens in the same time. Larger blocks with a larger token budget can thus be both faster and more accurate on long-context tasks than smaller blocks with a smaller budget. We evaluate two hybrid LLMs, Qwen3.5-4B and 35B-A3B, on one NVIDIA H200. Including the cost of building and using the index, whole-model wall-clock speedups over dense attention are 2.3-2.4x for prefill and 1.7-2.0x for decoding at 512K tokens, and 2.9-3.0x for prefill at 1M. The 35B-A3B model stays close to dense attention on five RULER tasks through 1M, on LongBench-v2 overall, and on MRCR through 256K. The 4B model loses accuracy mainly on multi-key retrieval, yet outperforms dense Qwen3.5-2B, whose latency is similar at 512K, on all three benchmarks.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.04313 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:32:37.436Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).