Metraipolis › archive › dc_aea1ff7c224a

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking — arXiv:2609.13141

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.13141
License (as archived)
CC0-1.0
Archived
2026-10-16T03:08:28.648Z
SHA-256 of the archived text
7a24330c36e09038a41dd199e792d43ff057ed1b244b17534f29bbd84bc929af
Stable URL of this page
https://metraipolis.com/documento/dc_aea1ff7c224a

How to cite

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking — arXiv:2609.13141 — arXiv (en), https://arxiv.org/abs/2609.13141 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aea1ff7c224a (Metraipolis archive dc_aea1ff7c224a, sha256 7a24330c36e09038a41dd199e792d43ff057ed1b244b17534f29bbd84bc929af)
BibTeX
@misc{dcaea1ff7c224a,
  title = {SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking — arXiv:2609.13141},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.13141},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aea1ff7c224a; sha256 7a24330c36e09038a41dd199e792d43ff057ed1b244b17534f29bbd84bc929af},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.13141. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context units (tokens or blocks) for each query. Existing trainable methods usually use a lightweight selector to score context units, followed by hard Top-K selection that blocks gradients from the language modeling loss. Consequently, these methods commonly distill layer-wise dense attention distributions. Although this encourages the selector to rank context units by dense attention weights in the original model, the ranking is not directly aligned with their impact on predictions under a fixed attention budget (i.e., the number of attended context units per query), potentially wasting the limited budget on less useful units. To address this misalignment, we propose Simple Attention Sparsification (SAS), a gated sparse attention mechanism that optimizes context ranking end-to-end with the language modeling loss. The key idea is to inject the selector's continuous scores into attention logits during training, allowing the loss to update the selector through standard backpropagation. We identify several choices crucial for this simple design to work well in practice: placing the gate inside the attention softmax in log form, using normalized softmax gates to calibrate historical context against the always-retained current block, and preserving continuous selector scores so the model learns relative priorities rather than only hard selections. To support long-sequence training, we implement a memory-efficient Triton kernel that integrates SAS into FlashAttention-style computation. Across reasoning, long-context understanding, and agentic tasks, SAS consistently outperforms trainable sparse attention baselines across attention budgets, with especially large gains under tight budgets, demonstrating more effective context ranking for downstream tasks.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-14 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.13141 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T03:02:47.560Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).