Metraipolis › archive › dc_ec4cdb1edcac
TurboPairFormer: Fast and Stable Protein Folding Model Training with an Optimized Triangle Attention Kernel — arXiv:2610.05854
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.05854
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T05:31:17.989Z
- SHA-256 of the archived text
747c70a3f2a8ba564dd42d561522034180b4798fd789b5d51f34e083855e5f8b- Stable URL of this page
https://metraipolis.com/documento/dc_ec4cdb1edcac
How to cite
TurboPairFormer: Fast and Stable Protein Folding Model Training with an Optimized Triangle Attention Kernel — arXiv:2610.05854 — arXiv, https://arxiv.org/abs/2610.05854 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ec4cdb1edcac (Metraipolis archive dc_ec4cdb1edcac, sha256 747c70a3f2a8ba564dd42d561522034180b4798fd789b5d51f34e083855e5f8b)
BibTeX
@misc{dcec4cdb1edcac,
title = {TurboPairFormer: Fast and Stable Protein Folding Model Training with an Optimized Triangle Attention Kernel — arXiv:2610.05854},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.05854},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ec4cdb1edcac; sha256 747c70a3f2a8ba564dd42d561522034180b4798fd789b5d51f34e083855e5f8b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05854. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Triangular attention is a core computation in AlphaFold3-style biomolecular models, with cubic cost in token count. Its shared pair bias adds a gradient reduction across attention slices to the usual reductions over queries and keys. The open-source backends we examine handle these reductions through repeated probability recomputation, floating-point atomics, or full score-gradient storage. Separately, computing the softmax backward correction from BF16-rounded forward outputs loses numerical precision. We present TurboPairFormer, a triangular attention implementation for NVIDIA Hopper GPUs that addresses both issues. Our key-tile-parallel backward algorithm recomputes each probability tile once for the query, key, value, and pair-bias gradients, using ordered partial reductions for deterministic accumulation without floating-point atomics or full score-gradient storage. Output-residual compensation retains a BF16 approximation of the output-rounding residual to compute the backward correction more accurately in FP32, without changing the BF16 output. With BF16 inputs at crop sizes 384, 640, and 768 and head dimensions 16 and 32, TurboPairFormer achieves the lowest mean query, key, and pair-bias gradient RMSE against an FP64 reference among the implementations compared in this paper. Residual compensation reduces these RMSE values by 28-47% in controlled ablations. All four gradients are bitwise identical across five repeated calls in all 600 input cases under fixed execution conditions. Integrated into OpenFold3 with our triangle multiplication kernels, TurboPairFormer achieves the lowest GPU computation time per optimizer step among the evaluated backend configurations on 16 H100 GPUs, with speedups of $1.73\times$ over OpenFold3's Triton backend and $1.13\times$ over cuEquivariance at crop size 768. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.05854 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T05:27:05.831Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).