Metraipolis › archive › dc_cab10a93f6fa

Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry — arXiv:2608.06849

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.06849
License (as archived)
CC0-1.0
Archived
2026-10-19T09:46:50.980Z
SHA-256 of the archived text
d42a9bdc64ef4eebaefcb9cd03517f6c85fc6241e2988f217cc49bd8ccd4f321
Stable URL of this page
https://metraipolis.com/documento/dc_cab10a93f6fa

How to cite

Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry — arXiv:2608.06849 — arXiv (en), https://arxiv.org/abs/2608.06849 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cab10a93f6fa (Metraipolis archive dc_cab10a93f6fa, sha256 d42a9bdc64ef4eebaefcb9cd03517f6c85fc6241e2988f217cc49bd8ccd4f321)
BibTeX
@misc{dccab10a93f6fa,
  title = {Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry — arXiv:2608.06849},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.06849},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cab10a93f6fa; sha256 d42a9bdc64ef4eebaefcb9cd03517f6c85fc6241e2988f217cc49bd8ccd4f321},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.06849. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Long-context LLM inference is bottlenecked by quadratic attention computation and growing KV-cache costs. Existing sparse attention and KV-compression methods typically decide which tokens or heads to preserve from runtime attention scores, observation windows, calibration prompts, or learned gates, making head diagnosis input-dependent and costly to deploy. We propose Autonomy-of-Heads (AoH), a data-free method that identifies retrieval and streaming heads from the spectral geometry of query-key projections. AoH defines the kernel attention operator $M_h = W_K^{h\top}W_Q^h$ and uses its effective-rank as a weight-space measure of head function: concentrated spectra indicate a small number of dominant query-key matching directions and are associated with retrieval heads, whereas diffuse spectra indicate the absence of a dominant global matching direction and are associated with streaming heads. We further derive an efficient $d_\text{head}$-dimensional computation that avoids constructing the full $d_\text{model}\times d_\text{model}$ matrix. We conducted extensive experiments across models demonstrating that at 50\% sparsity, AoH retains 96.5\% of Full Attention performance on average while reducing prefill and decode latency by up to 41.4\% and 66.0\%, respectively, and KV-cache memory by 50.0\% at 256K tokens.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-10 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.06849 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T09:46:37.222Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).