Metraipolis › archive › dc_bee29ebff36c

Coarse Semantic Blocks for Efficient Training-Free Sparse Attention — arXiv:2611.04313

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.04313
License (as archived): CC0-1.0
Archived: 2026-11-04T21:39:59.213Z
SHA-256 of the archived text: 76b7f49e7f731759ff7b4d236336edbca72f3e1fd3f3e7c70d288daeda0b9bb4
Stable URL of this page: https://metraipolis.com/documento/dc_bee29ebff36c

How to cite

Coarse Semantic Blocks for Efficient Training-Free Sparse Attention — arXiv:2611.04313 — arXiv (en), https://arxiv.org/abs/2611.04313 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bee29ebff36c (Metraipolis archive dc_bee29ebff36c, sha256 76b7f49e7f731759ff7b4d236336edbca72f3e1fd3f3e7c70d288daeda0b9bb4)

BibTeX

@misc{dcbee29ebff36c,
  title = {Coarse Semantic Blocks for Efficient Training-Free Sparse Attention — arXiv:2611.04313},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04313},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bee29ebff36c; sha256 76b7f49e7f731759ff7b4d236336edbca72f3e1fd3f3e7c70d288daeda0b9bb4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04313. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We introduce a training-free sparse attention method that builds a prompt-specific key index to accelerate both prefill and decoding. The method clusters the keys into large semantic blocks that group tokens from noncontiguous positions. Each query position selects blocks by scoring their centroids and attends to the original keys and values within them. Queries that select the same block are processed together, so prefill stays fast even though each query chooses its own blocks. Large blocks make centroid scoring and this grouped processing cheap, so each query can attend to more tokens in the same time. Larger blocks with a larger token budget can thus be both faster and more accurate on long-context tasks than smaller blocks with a smaller budget. We evaluate two hybrid LLMs, Qwen3.5-4B and 35B-A3B, on one NVIDIA H200. Including the cost of building and using the index, whole-model wall-clock speedups over dense attention are 2.3-2.4x for prefill and 1.7-2.0x for decoding at 512K tokens, and 2.9-3.0x for prefill at 1M. The 35B-A3B model stays close to dense attention on five RULER tasks through 1M, on LongBench-v2 overall, and on MRCR through 256K. The 4B model loses accuracy mainly on multi-key retrieval, yet outperforms dense Qwen3.5-2B, whose latency is similar at 512K, on all three benchmarks.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.04313 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T21:35:49.663Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).