Metraipolis › archive › dc_fed5ebe28bcc

FlashAttention for Scalable Vector Architectures — arXiv:2608.18656

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.18656
License (as archived): CC0-1.0
Archived: 2026-10-18T13:53:20.840Z
SHA-256 of the archived text: 5454be6abe24884f6eeb3376a60ee9b5e9658d233195cea5a6c078899587b966
Stable URL of this page: https://metraipolis.com/documento/dc_fed5ebe28bcc

How to cite

FlashAttention for Scalable Vector Architectures — arXiv:2608.18656 — arXiv (en), https://arxiv.org/abs/2608.18656 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fed5ebe28bcc (Metraipolis archive dc_fed5ebe28bcc, sha256 5454be6abe24884f6eeb3376a60ee9b5e9658d233195cea5a6c078899587b966)

BibTeX

@misc{dcfed5ebe28bcc,
  title = {FlashAttention for Scalable Vector Architectures — arXiv:2608.18656},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.18656},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fed5ebe28bcc; sha256 5454be6abe24884f6eeb3376a60ee9b5e9658d233195cea5a6c078899587b966},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.18656. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Inference with transformer models on CPUs is increasingly important, especially for Small Language Models (SLMs), where vector architectures are emerging as a promising execution substrate. The attention module is a major bottleneck due to high memory bandwidth requirements; FlashAttention mitigates this by fusing operations to improve data locality and reduce intermediate memory traffic. In this paper, we present FlashAttention-V, a blocked FlashAttention for scalable vector architectures that adapts efficiently from short to very long vectors by exploiting parallelism across attention heads, inter-head packing to enable efficient utilization of vector lengths beyond the head dimension, and improving vector register utilization and memory access locality. We integrate FlashAttention-V into ggml within this http URL and evaluate it on TinyLlama, Llama 3.2, Qwen2.5, and Pythia-410M using gem5 and a Banana Pi BPI-F3. On the Banana Pi BPI-F3, we confirm that loop reordering and loop unrolling across attention heads are effective optimization principles, scaling performance gains with larger models and most pronounced with short contexts and during decoding. Simulation-based analysis shows that FlashAttention-V achieves 22x-42x speedup over scalar FlashAttention at 512-bit VL in prefill, with an additional 2x-2.5x gain scaling to 64 lanes and 4096-bit VL. During decode, FlashAttention-V achieves 8x-11x speedup using 512-bit vector lengths over scalar FlashAttention, with performance showing diminishing sensitivity to vector width and lane count due to single-token, memory-bound execution. We further identify structural bottlenecks in Q8_0 quantized linear layers that limit arithmetic amortization under long-vector execution, consistent across RVV and Arm SVE, indicating that current quantization formats pose a fundamental challenge to long-vector scalability.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-20 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2608.18656 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T13:50:58.923Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).