Metraipolis › archive › dc_227aa08f0926

LazyFlash: Forward-Dense, Backward-Sparse Attention for Accelerated Long-Context Fine-Tuning — arXiv:2610.31911

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.31911
License (as archived)
CC0-1.0
Archived
2026-10-26T19:52:54.418Z
SHA-256 of the archived text
1265cca65ecdbeb794270c982567c50bf8bc7ad1a5294ff760fc05a81d7f5919
Stable URL of this page
https://metraipolis.com/documento/dc_227aa08f0926

How to cite

LazyFlash: Forward-Dense, Backward-Sparse Attention for Accelerated Long-Context Fine-Tuning — arXiv:2610.31911 — arXiv (en), https://arxiv.org/abs/2610.31911 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_227aa08f0926 (Metraipolis archive dc_227aa08f0926, sha256 1265cca65ecdbeb794270c982567c50bf8bc7ad1a5294ff760fc05a81d7f5919)
BibTeX
@misc{dc227aa08f0926,
  title = {LazyFlash: Forward-Dense, Backward-Sparse Attention for Accelerated Long-Context Fine-Tuning — arXiv:2610.31911},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.31911},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_227aa08f0926; sha256 1265cca65ecdbeb794270c982567c50bf8bc7ad1a5294ff760fc05a81d7f5919},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31911. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As large language models (LLMs) process increasingly long sequences, attention becomes a major computational bottleneck. Sparse attention mostly targets inference, yet many methods still train densely or cannot support sparse training, and their added complexity makes sparsity hard to turn into wall-clock speedups, leaving dense attention widely used in practice. This paper introduces LazyFlash, a forward-dense, backward-sparse attention method built on the observation that attention training is dominated by the backward pass. LazyFlash keeps the forward pass exactly dense, so the trained model remains a standard dense model that existing systems such as vLLM can serve without any change. For the backward pass, LazyFlash proposes a gradient-aware block sparsification design that ranks query blocks by their attention-output gradient norms and skips those contributing negligibly to parameter updates, turning the dense backward into a block-sparse one. A gradient-mass budget adapts the amount of skipped computation to each head and training step, while importance sampling preserves the dense gradient in expectation. These designs are implemented inside the tiled FlashAttention backward kernel, where skipped blocks are never loaded or computed, translating gradient sparsity into real wall-clock acceleration. Experiments on long-context text and speech LLMs show that LazyFlash maintains downstream performance while achieving up to 5.7x attention-backward speedup over FlashAttention and 1.7x end-to-end training speedup.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.31911 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-26T19:45:48.335Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).