Metraipolis › archive › dc_4635c275a397

FLAMIA: FLAsh MIcroscaling Attention for Fast and Causally Safe Training on Blackwell — arXiv:2611.03344

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.03344
License (as archived): CC0-1.0
Archived: 2026-11-03T11:23:12.236Z
SHA-256 of the archived text: 8b46a2c5ead3cabaf311afbad7662877cc4bc73b59afc2bbcf1353f66ca7b6f5
Stable URL of this page: https://metraipolis.com/documento/dc_4635c275a397

How to cite

FLAMIA: FLAsh MIcroscaling Attention for Fast and Causally Safe Training on Blackwell — arXiv:2611.03344 — arXiv (en), https://arxiv.org/abs/2611.03344 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4635c275a397 (Metraipolis archive dc_4635c275a397, sha256 8b46a2c5ead3cabaf311afbad7662877cc4bc73b59afc2bbcf1353f66ca7b6f5)

BibTeX

@misc{dc4635c275a397,
  title = {FLAMIA: FLAsh MIcroscaling Attention for Fast and Causally Safe Training on Blackwell — arXiv:2611.03344},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03344},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4635c275a397; sha256 8b46a2c5ead3cabaf311afbad7662877cc4bc73b59afc2bbcf1353f66ca7b6f5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03344. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Attention remains a substantial computational cost in Transformer training, particularly at long context lengths. NVIDIA Blackwell GPUs provide high-throughput low-precision matrix multiplication, but exploiting it in attention requires addressing quantization overhead, causal information leakage through shared scales, and numerical inconsistencies between the forward and backward passes. We present FLAMIA, a family of attention kernels for Blackwell with MXFP8 and mixed NVFP4/FP8 training paths. Our design combines token-block scaling that reuses quantized operands across matrix orientations, targeted safeguards against scale-induced leakage, and consistent forward and backward computations. Fused quantization and architecture-specific kernel optimizations reduce overhead, while document-local scaling and workload-aware scheduling support variable-length documents directly in packed layouts without repacking. Across various causal packed-document workloads, the MXFP8 path accelerates the combined forward and backward attention call, including quantization, by factors of $1.10$--$1.40\times$ on B200 and $1.06$--$1.42\times$ on B300 relative to the fastest evaluated baseline for each workload. We validate training stability and quality preservation on pretraining Llama-style models. These results show how jointly designing quantization, training numerics, and document scheduling translates low-bit hardware throughput into practical attention acceleration.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.03344 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T11:02:37.694Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).