Metraipolis › archive › dc_80f19bf2157f
Fast and Exact Relative-Position Attention via Structure-Aware Kernels — arXiv:2611.01182
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.01182
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:12:47.846Z
- SHA-256 of the archived text
6581246bc05f3fdf84c191adbeee667ba0b8d17aa68a192829754d290e3d6ad8- Stable URL of this page
https://metraipolis.com/documento/dc_80f19bf2157f
How to cite
Fast and Exact Relative-Position Attention via Structure-Aware Kernels — arXiv:2611.01182 — arXiv (en), https://arxiv.org/abs/2611.01182 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_80f19bf2157f (Metraipolis archive dc_80f19bf2157f, sha256 6581246bc05f3fdf84c191adbeee667ba0b8d17aa68a192829754d290e3d6ad8)
BibTeX
@misc{dc80f19bf2157f,
title = {Fast and Exact Relative-Position Attention via Structure-Aware Kernels — arXiv:2611.01182},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.01182},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_80f19bf2157f; sha256 6581246bc05f3fdf84c191adbeee667ba0b8d17aa68a192829754d290e3d6ad8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01182. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Relative-position attention mechanisms offer compelling alternatives to RoPE across various transformer architectures, delivering superior performance in domains such as audio. These mechanisms typically modify attention scores by adding content-position or bias-position terms. Unlike RoPE, which encodes position via per-token rotations directly on the queries and keys, relative-position terms generally require pairwise offset-dependent computation or lookups, complicating their integration into fused attention kernels and introducing severe memory, bandwidth, and compute overheads as sequences scale. To address this bottleneck, we propose an exact attention algorithm that incorporates additive relative-position terms into FlashAttention-style tiled computation. Our method (1) uses offset-aware tiling to avoid redundant indexing and memory accesses, (2) computes only the necessary key/query-relation products and reuses them across token pairs in both forward and backward passes, and (3) uses offset-based gradient sharding and aggregation, eliminating atomic contention. We provide a unified, efficient Triton implementation across diverse relative-position attention mechanisms and evaluate it against naive and specialized implementations. For 8K-token sequences, our method speeds up the forward (backward) pass by 1.8-6.4$\times$ (2.0-21$\times$) and reduces peak memory up to 2.9-128$\times$ (1.4-54$\times$) over the strongest baseline for each mechanism, enabling long-context end-to-end transcription that is 1.8$\times$ faster and uses 2.4$\times$ less memory. Our method thus provides a drop-in, hardware-efficient solution for scaling relative-position attention transformers. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2611.01182 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T11:00:07.173Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).