Metraipolis › archive › dc_77355dcbeea8

Validating Memory-Optimal Transformer Kernels on Real Hardware: From Formal Derivation to Measured Performance Across Two HPC Clusters — arXiv:2609.33916

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.33916
License (as archived): CC0-1.0
Archived: 2026-10-07T21:01:54.306Z
SHA-256 of the archived text: 0de7394880ba5825df5522387f5fd0c557259081811030602c25b662b2eac2f2
Stable URL of this page: https://metraipolis.com/documento/dc_77355dcbeea8

How to cite

Validating Memory-Optimal Transformer Kernels on Real Hardware: From Formal Derivation to Measured Performance Across Two HPC Clusters — arXiv:2609.33916 — arXiv (en), https://arxiv.org/abs/2609.33916 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_77355dcbeea8 (Metraipolis archive dc_77355dcbeea8, sha256 0de7394880ba5825df5522387f5fd0c557259081811030602c25b662b2eac2f2)

BibTeX

@misc{dc77355dcbeea8,
  title = {Validating Memory-Optimal Transformer Kernels on Real Hardware: From Formal Derivation to Measured Performance Across Two HPC Clusters — arXiv:2609.33916},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.33916},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_77355dcbeea8; sha256 0de7394880ba5825df5522387f5fd0c557259081811030602c25b662b2eac2f2},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33916. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We validate memory-optimal cost functions for transformer kernels derived via the Mathematics of Arrays (MoA). Companion Papers I-IV formally derive kernels for attention forward, backward, fused forward+backward, decode, and the complete block (RMSNorm, gated MLP) as a hardware-independent specification (DNF) transformed to a machine-specific realization (ONF) via gamma, with verification to machine precision against PyTorch. This paper checks those predictions against measured performance on two HPC clusters (Purdue Anvil, NCSA Delta) across CPU and GPU. Three results stand out. (1) We identify and fix a GPU regression: fusing forward+backward, proven to avoid materializing an O(n^2) intermediate, initially ran slower than naive on GPU due to atomic contention. Profiling confirmed 2.00x more atomic instructions; a targeted ONF rewrite reversed it, yielding up to 2.5x speedup. (2) Identical derivations produce markedly different real costs by topology: 535x NUMA-locality penalty on one cluster vs <3x oversubscription on another, showing optimal deployment is a function of the machine's array structure. (3) We report a partially resolved anomaly: identical denotational computations run faster in C than Fortran on CPU but faster in Fortran than C on GPU, narrowed to one dominant kernel and one memory-latency stall mechanism (3.17x time gap matches 3.35x stall gap). We treat hardware-specific optimization as a routine ONF rewrite with fixed, verified DNF, a candidate methodology for scaling AI onto evolving hardware without re-deriving correctness.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.33916 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:02.784Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).