Metraipolis › archive › dc_6d798ae361da

Does Transolver really need a Transformer? — arXiv:2609.32525

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.32525
License (as archived)
CC0-1.0
Archived
2026-10-07T22:36:40.130Z
SHA-256 of the archived text
9d1bd14d8f6d16f49cd5b4c4056a62c111ef03ec72306fb587549fb2e079c9d6
Stable URL of this page
https://metraipolis.com/documento/dc_6d798ae361da

How to cite

Does Transolver really need a Transformer? — arXiv:2609.32525 — arXiv (en), https://arxiv.org/abs/2609.32525 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6d798ae361da (Metraipolis archive dc_6d798ae361da, sha256 9d1bd14d8f6d16f49cd5b4c4056a62c111ef03ec72306fb587549fb2e079c9d6)
BibTeX
@misc{dc6d798ae361da,
  title = {Does Transolver really need a Transformer? — arXiv:2609.32525},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32525},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6d798ae361da; sha256 9d1bd14d8f6d16f49cd5b4c4056a62c111ef03ec72306fb587549fb2e079c9d6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32525. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The widely used Transolver family of neural operators is based on physics-attention, which softly assigns the points of an unstructured mesh to a small number of slices, applies self-attention among the resulting tokens, and broadcasts the result back to the points. We provide a comprehensive empirical and theoretical analysis to elucidate the mechanisms which are responsible for model performance. To this end, we perform careful ablations on a challenging suite of nine 3D fluid dynamics benchmarks to find that replacing token attention with a constant linear map does not affect the accuracy. Thus, Transolver does not need a Transformer at all. However, removing the global mixing (slicing/deslicing) or doing it only once leads to performance collapse. We leverage the theory of averaging neural operators to explain and corroborate our findings by showing that just slicing/deslicing, in conjunction with pointwise MLPs, already suffices for universal approximation of continuous operators and attention is redundant in this context. Finally, we provide a novel FlashAttention-style efficient implementation of the key slicing/deslicing module of Transolver. This flashslice kernel streams slice and deslice over the points without materializing the heavy slice-weight tensor, while reproducing the best available implementation to floating point error. At the same time, it leads to very significant memory and compute savings, particularly at large slice counts.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.32525 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:27.709Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).