Metraipolis › archive › dc_bc35e2c65386

SparseGDN: Chunk Sparse in Linear Attention — arXiv:2610.41420

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.41420
License (as archived)
CC0-1.0
Archived
2026-11-02T07:40:41.959Z
SHA-256 of the archived text
393fdd7c8af4e60003d1e4764d8c64e2496eed9e8b1fc2c1c8165ed9c61809cc
Stable URL of this page
https://metraipolis.com/documento/dc_bc35e2c65386

How to cite

SparseGDN: Chunk Sparse in Linear Attention — arXiv:2610.41420 — arXiv (en), https://arxiv.org/abs/2610.41420 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bc35e2c65386 (Metraipolis archive dc_bc35e2c65386, sha256 393fdd7c8af4e60003d1e4764d8c64e2496eed9e8b1fc2c1c8165ed9c61809cc)
BibTeX
@misc{dcbc35e2c65386,
  title = {SparseGDN: Chunk Sparse in Linear Attention — arXiv:2610.41420},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41420},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bc35e2c65386; sha256 393fdd7c8af4e60003d1e4764d8c64e2496eed9e8b1fc2c1c8165ed9c61809cc},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41420. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Gated Delta Networks (GDNs) offer linear-time sequence modeling, yet long-context prefill remains costly due to dense intra-chunk computation, recurrent-state updates, and intermediate state storage. We introduce SparseGDN, a hardware-aware framework that combines structured intra-chunk approximation with input-dependent recurrent-event selection while producing outputs for every token. A block-adjacent solver reduces local Delta-solve computation, while a dynamic selector allocates recurrent-update budgets across head pairs. The GPU implementation materializes updates only for selected events and reconstructs token outputs using compact state snapshots and group-local interactions, reducing computation and memory traffic. We evaluate SparseGDN on Qwen3.5-4B and Qwen3.5-9B across RULER tasks and context lengths from 4K to 128K. Retaining approximately 29% of recurrent events on average, SparseGDN achieves 1.50-1.94x GDN-core prefill speedups over native FLA kernels, including online selection overhead.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.41420 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T07:37:00.578Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).