Metraipolis › archive › dc_653eba770b85

Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention — arXiv:2610.42147

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.42147
License (as archived)
CC0-1.0
Archived
2026-11-03T08:02:16.000Z
SHA-256 of the archived text
d9a150af48e19341b00e923daa21dcf24cfaed6663ba0ba0e9742cdccdc10abf
Stable URL of this page
https://metraipolis.com/documento/dc_653eba770b85

How to cite

Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention — arXiv:2610.42147 — arXiv (en), https://arxiv.org/abs/2610.42147 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_653eba770b85 (Metraipolis archive dc_653eba770b85, sha256 d9a150af48e19341b00e923daa21dcf24cfaed6663ba0ba0e9742cdccdc10abf)
BibTeX
@misc{dc653eba770b85,
  title = {Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention — arXiv:2610.42147},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.42147},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_653eba770b85; sha256 d9a150af48e19341b00e923daa21dcf24cfaed6663ba0ba0e9742cdccdc10abf},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.42147. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. However, at ultra-long context (above 1M tokens), full attention is prohibitively expensive. A promising approach is sparse attention, in which a dedicated indexer selects a subset of keys for the attention. Yet current methods of indexing either still have quadratic cost in context size or lose accuracy at ultra-long context scale. We address this with Latent Sparse Attention (LSA), an efficient attention method that maintains accuracy at ultra-long context. In LSA, keys are hierarchically clustered in a learned latent space, and queries score the hierarchy of clusters instead of scoring every key, making prefill, decode and training faster at long context. We propose two variants, Distilled-LSA (D-LSA), which can be applied to pre-trained models via distillation, and Native-LSA (N-LSA), which is trained natively on the next-token loss alone. On GLM-5.3-FLASH (QWEN-3.8-27B) at 10M tokens, D-LSA speeds up one attention layer by 22.7x (72.4x) in prefill and 4.6x (42.5x) in decode over its original sparse attention (full attention), while keeping similar accuracy across a variety of tasks. N-LSA matches the accuracy of full attention with a training speedup of 1.6x per layer at 128k tokens. LSA offers practical and efficient attention for models to benefit from ultra-long context.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.42147 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:59:38.755Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).