Metraipolis › archive › dc_fecd3bc6d719

MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs — arXiv:2609.33135

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.33135
License (as archived)
CC0-1.0
Archived
2026-10-07T22:12:05.298Z
SHA-256 of the archived text
19068da89341709d10f4502259fe081355bc38ffdf432a0abd8f55e502bd0f0e
Stable URL of this page
https://metraipolis.com/documento/dc_fecd3bc6d719

How to cite

MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs — arXiv:2609.33135 — arXiv (en), https://arxiv.org/abs/2609.33135 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fecd3bc6d719 (Metraipolis archive dc_fecd3bc6d719, sha256 19068da89341709d10f4502259fe081355bc38ffdf432a0abd8f55e502bd0f0e)
BibTeX
@misc{dcfecd3bc6d719,
  title = {MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs — arXiv:2609.33135},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.33135},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fecd3bc6d719; sha256 19068da89341709d10f4502259fe081355bc38ffdf432a0abd8f55e502bd0f0e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33135. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Long-context LLM inference pushes modern GPU serving stacks into an attention-bound regime, where both compute and memory are dominated by the softmax-GEMM pipeline. On NVIDIA Blackwell GPUs, FP4 Tensor Cores offer high matmul throughput, but we find that fully FP4 attention often fails to translate this throughput into end-to-end speedups due to non-matmul costs: online quantization after softmax, tensor/shared-memory data movement, and contention on the softmax path. We present MpFA, a training-free FlashAttention kernel optimized for Blackwell. Guided by hardware characterization, MpFA uses mixed precision: NVFP4 for QK and FP8 for PV (QK4PV8). This preserves low-bit QK throughput while avoiding the conversion and scaling overheads of FP4 PV. To recover accuracy without further stressing the softmax pipeline, MpFA introduces rank-one smoothing compensation implemented as an additional Tensor Core MMA. MpFA further improves performance with a fine-grained asynchronous pipeline, tensor-memory reuse, and adaptive parallel partitioning across prefill and decode. On an NVIDIA B200 and across 16K-128K contexts, MpFA improves prefill throughput over state-of-the-art BF16/FP8 baselines and increases end-to-end output throughput by 2.81$\times$ over BF16 FA4 across Llama-3.1-8B and Qwen3-14B. Across five benchmark suites and two models, rank-one compensation recovers 62.5% of the accuracy loss with about 2.0% kernel overhead.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-29 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2609.33135 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T21:53:13.667Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).