Metraipolis › archive › dc_6da43f70d41a

When and Why to Use TMA: A Workload-Grounded Characterization of Tensor Memory Acceleration on NVIDIA Hopper GPUs — arXiv:2610.34907

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.34907
License (as archived)
CC0-1.0
Archived
2026-10-27T08:14:27.764Z
SHA-256 of the archived text
4627d90034dfd2a281947484e7da26eb6193fc6489c557cd059a68d1ab461dda
Stable URL of this page
https://metraipolis.com/documento/dc_6da43f70d41a

How to cite

When and Why to Use TMA: A Workload-Grounded Characterization of Tensor Memory Acceleration on NVIDIA Hopper GPUs — arXiv:2610.34907 — arXiv (en), https://arxiv.org/abs/2610.34907 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6da43f70d41a (Metraipolis archive dc_6da43f70d41a, sha256 4627d90034dfd2a281947484e7da26eb6193fc6489c557cd059a68d1ab461dda)
BibTeX
@misc{dc6da43f70d41a,
  title = {When and Why to Use TMA: A Workload-Grounded Characterization of Tensor Memory Acceleration on NVIDIA Hopper GPUs — arXiv:2610.34907},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.34907},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6da43f70d41a; sha256 4627d90034dfd2a281947484e7da26eb6193fc6489c557cd059a68d1ab461dda},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.34907. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The Tensor Memory Accelerator (TMA), introduced on NVIDIA Hopper, is used pervasively in production GPU kernels yet its usage is conspicuously selective: expert kernel developers in CUTLASS, FlashAttention-3, and FlashInfer apply TMA to some tensor operands within a kernel but fall back to this http URL , direct global loads, or distributed shared memory (DSM) for others. Vendor documentation describes the API, and recent microbenchmarking work characterizes TMA throughput in isolation, but neither answers the question practitioners actually face: given a memory operand in a kernel, should it use TMA, and if so, with or without cluster multicast? We address this gap with a workload-grounded characterization study on H100. We catalog the TMA, multicast, DSM, and this http URL choices made inside FlashAttention-3 forward, isolate the discriminating features through targeted microbenchmarks (transfer-size crossover, multicast scaling), and produce rules of thumb for the load-mechanism choice on Hopper. We further compare TMA multicast against DSM as competing cluster-broadcast primitives, and isolate the FlashAttention-3 paged-key-value (KV) this http URL dispatch on H100 across page size, datatype, model shape, and execution phase. Our results yield concrete, predictive guidance on when TMA pays off and when multicast helps or hurts on Hopper, and we extend the analysis to server Blackwell (SM100) via a CUTLASS GEMM study on B200 that shows the qualitative dispatch rules carry across generations while absolute crossovers shift with tensor-core and bandwidth scaling.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Hardware Architecture (cs.AR).
Pagina del lavoro: https://arxiv.org/abs/2610.34907 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:56:15.148Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).