Metraipolis › archive › dc_d4bfd76fe063
SparseGDN: Chunk Sparse in Linear Attention — arXiv:2610.41420
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.41420
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T07:52:35.753Z
- SHA-256 of the archived text
3cdf7dcc09080ebda8f5ffa36f722cc96368756767db0fa11e55cbe86eae53de- Stable URL of this page
https://metraipolis.com/documento/dc_d4bfd76fe063
How to cite
SparseGDN: Chunk Sparse in Linear Attention — arXiv:2610.41420 — arXiv (en), https://arxiv.org/abs/2610.41420 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d4bfd76fe063 (Metraipolis archive dc_d4bfd76fe063, sha256 3cdf7dcc09080ebda8f5ffa36f722cc96368756767db0fa11e55cbe86eae53de)
BibTeX
@misc{dcd4bfd76fe063,
title = {SparseGDN: Chunk Sparse in Linear Attention — arXiv:2610.41420},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.41420},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d4bfd76fe063; sha256 3cdf7dcc09080ebda8f5ffa36f722cc96368756767db0fa11e55cbe86eae53de},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41420. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Gated Delta Networks (GDNs) offer linear-time sequence modeling, yet long-context prefill remains costly due to dense intra-chunk computation, recurrent-state updates, and intermediate state storage. We introduce SparseGDN, a hardware-aware framework that combines structured intra-chunk approximation with input-dependent recurrent-event selection while producing outputs for every token. A block-adjacent solver reduces local Delta-solve computation, while a dynamic selector allocates recurrent-update budgets across head pairs. The GPU implementation materializes updates only for selected events and reconstructs token outputs using compact state snapshots and group-local interactions, reducing computation and memory traffic. We evaluate SparseGDN on Qwen3.5-4B and Qwen3.5-9B across RULER tasks and context lengths from 4K to 128K. Retaining approximately 29% of recurrent events on average, SparseGDN achieves 1.50-1.94x GDN-core prefill speedups over native FLA kernels, including online selection overhead. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.41420 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T07:49:31.160Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).