Metraipolis › archive › dc_380f2ff9f4da

AttnFuse: A Composable DSL for Compiling Attentions to Fused GPU Kernels — arXiv:2609.13612

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.13612
License (as archived)
CC0-1.0
Archived
2026-10-16T00:10:25.091Z
SHA-256 of the archived text
220fdd671236c364a85ecc2746bef379dc97b8b58d16442065efd5e56ce795d7
Stable URL of this page
https://metraipolis.com/documento/dc_380f2ff9f4da

How to cite

AttnFuse: A Composable DSL for Compiling Attentions to Fused GPU Kernels — arXiv:2609.13612 — arXiv (en), https://arxiv.org/abs/2609.13612 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_380f2ff9f4da (Metraipolis archive dc_380f2ff9f4da, sha256 220fdd671236c364a85ecc2746bef379dc97b8b58d16442065efd5e56ce795d7)
BibTeX
@misc{dc380f2ff9f4da,
  title = {AttnFuse: A Composable DSL for Compiling Attentions to Fused GPU Kernels — arXiv:2609.13612},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.13612},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_380f2ff9f4da; sha256 220fdd671236c364a85ecc2746bef379dc97b8b58d16442065efd5e56ce795d7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.13612. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Modern AI systems are built on the Transformer architecture, whose core operation, attention, accounts for the majority of computation and memory cost. Researchers continually propose new attention variants to improve quality, efficiency, or context length, but each variant currently requires expert-written GPU code to run at usable speeds. PyTorch's recent flex\_attention lets researchers describe custom attention patterns in Python and compile them to fused kernels, but its design is limited to modifications applied after the central matrix multiplication, excluding Rotary Position Embedding (RoPE), the positional encoding used by every major LLM. We introduce AttnFuse, a small DSL for attention that makes pre-multiplication transformations like RoPE first-class operations. Researchers compose ten high-level building blocks to describe a variant, and AttnFuse's compiler emits a single fused GPU kernel for the entire computation. On an RTX 3090, AttnFuse achieves a 2.10$\times$ speedup over flex\_attention on the RoPE+causal pattern. On an H100, it runs a full Llama-3-8B training step within 5\% of PyTorch's hand-tuned backend. Our investigation reveals the Rotation Calculus: whether to fuse RoPE or apply it separately depends on the GPU's compute-to-bandwidth ratio, with a derived crossover that matches measurement. AttnFuse demonstrates that a small, attention-specific compiler can close the gap between flexible research code and production kernels.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.13612 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T23:54:14.624Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).