Metraipolis › archive › dc_ca16a40831bd

Attend at the Right Resolution: Head-Adaptive Sparse Attention via Query-Key Interaction Structure — arXiv:2611.09700

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.09700
License (as archived)
CC0-1.0
Archived
2026-11-09T22:34:13.549Z
SHA-256 of the archived text
eaab18093910e8a76cdb7316fb83300554683d3131c582fc017a2c9b79cdb33c
Stable URL of this page
https://metraipolis.com/documento/dc_ca16a40831bd

How to cite

Attend at the Right Resolution: Head-Adaptive Sparse Attention via Query-Key Interaction Structure — arXiv:2611.09700 — arXiv (en), https://arxiv.org/abs/2611.09700 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ca16a40831bd (Metraipolis archive dc_ca16a40831bd, sha256 eaab18093910e8a76cdb7316fb83300554683d3131c582fc017a2c9b79cdb33c)
BibTeX
@misc{dcca16a40831bd,
  title = {Attend at the Right Resolution: Head-Adaptive Sparse Attention via Query-Key Interaction Structure — arXiv:2611.09700},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.09700},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ca16a40831bd; sha256 eaab18093910e8a76cdb7316fb83300554683d3131c582fc017a2c9b79cdb33c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.09700. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Attention is a major scalability bottleneck in large language models (LLMs), with its computational cost and key-value (KV) memory traffic becoming increasingly prohibitive as context length grows. Sparse attention addresses this bottleneck by restricting each query to a subset of the context, but its effectiveness hinges on identifying relevant regions both cheaply and reliably. Existing training-free methods primarily rely on runtime relevance estimates, with selection dimensionality and granularity typically governed by fixed heuristics rather than the intrinsic interaction structure of pretrained attention heads. We take a different perspective: the query-key operator of each pretrained head defines a characteristic response structure that determines the resolution needed for sparse selection, both across feature directions and across context tokens. Building on this insight, we propose Head-Adaptive Sparse Attention (HASA), a training-free sparse attention framework that jointly adapts selection resolution along these two dimensions. Along the feature dimension, the singular spectrum of each head's query-key operator determines an effective rank, yielding a compact, head-specific response space. Along the sequence dimension, neighboring regions are progressively merged according to the directional coherence of their token responses in this same space, while heterogeneous regions remain separate. Exact attention is then computed using the original full-dimensional KV states of the selected regions. HASA captures 88.1% of dense-attention mass and, compared with UNIQUE, a strong recent sparse-attention baseline, improves accuracy by up to 3.47 and 2.57 points on RULER and LongBench-Pro, respectively, and accelerates self-attention by up to 1.32x, while achieving up to 14.07x speedup over dense attention.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.09700 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-09T22:30:47.289Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).