Metraipolis › archive › dc_dba509e910ce

Fast and Exact Relative-Position Attention via Structure-Aware Kernels — arXiv:2611.01182

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.01182
License (as archived)
CC0-1.0
Archived
2026-11-03T18:26:48.390Z
SHA-256 of the archived text
00b3c1808946ae16b5caae6272e11054fcd27183bbf6da64908a8ab5889fcc86
Stable URL of this page
https://metraipolis.com/documento/dc_dba509e910ce

How to cite

Fast and Exact Relative-Position Attention via Structure-Aware Kernels — arXiv:2611.01182 — arXiv (en), https://arxiv.org/abs/2611.01182 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dba509e910ce (Metraipolis archive dc_dba509e910ce, sha256 00b3c1808946ae16b5caae6272e11054fcd27183bbf6da64908a8ab5889fcc86)
BibTeX
@misc{dcdba509e910ce,
  title = {Fast and Exact Relative-Position Attention via Structure-Aware Kernels — arXiv:2611.01182},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01182},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dba509e910ce; sha256 00b3c1808946ae16b5caae6272e11054fcd27183bbf6da64908a8ab5889fcc86},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01182. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Relative-position attention mechanisms offer compelling alternatives to RoPE across various transformer architectures, delivering superior performance in domains such as audio. These mechanisms typically modify attention scores by adding content-position or bias-position terms. Unlike RoPE, which encodes position via per-token rotations directly on the queries and keys, relative-position terms generally require pairwise offset-dependent computation or lookups, complicating their integration into fused attention kernels and introducing severe memory, bandwidth, and compute overheads as sequences scale. To address this bottleneck, we propose an exact attention algorithm that incorporates additive relative-position terms into FlashAttention-style tiled computation. Our method (1) uses offset-aware tiling to avoid redundant indexing and memory accesses, (2) computes only the necessary key/query-relation products and reuses them across token pairs in both forward and backward passes, and (3) uses offset-based gradient sharding and aggregation, eliminating atomic contention. We provide a unified, efficient Triton implementation across diverse relative-position attention mechanisms and evaluate it against naive and specialized implementations. For 8K-token sequences, our method speeds up the forward (backward) pass by 1.8-6.4$\times$ (2.0-21$\times$) and reduces peak memory up to 2.9-128$\times$ (1.4-54$\times$) over the strongest baseline for each mechanism, enabling long-context end-to-end transcription that is 1.8$\times$ faster and uses 2.4$\times$ less memory. Our method thus provides a drop-in, hardware-efficient solution for scaling relative-position attention transformers.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2611.01182 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:10:22.851Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).