Metraipolis › archive › dc_cdaf8ca06d5c
Geometry Matters in Packed 3D Attention — arXiv:2610.38694
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.38694
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T06:12:52.838Z
- SHA-256 of the archived text
f87c7aa1c9587ae96f8866cefa1516a05864cbec15280d69a82bfffe39381906- Stable URL of this page
https://metraipolis.com/documento/dc_cdaf8ca06d5c
How to cite
Geometry Matters in Packed 3D Attention — arXiv:2610.38694 — arXiv (en), https://arxiv.org/abs/2610.38694 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cdaf8ca06d5c (Metraipolis archive dc_cdaf8ca06d5c, sha256 f87c7aa1c9587ae96f8866cefa1516a05864cbec15280d69a82bfffe39381906)
BibTeX
@misc{dccdaf8ca06d5c,
title = {Geometry Matters in Packed 3D Attention — arXiv:2610.38694},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.38694},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cdaf8ca06d5c; sha256 f87c7aa1c9587ae96f8866cefa1516a05864cbec15280d69a82bfffe39381906},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.38694. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Packing a 3D volume into a token sequence is not merely long-sequence modeling. Once the volume is packed, sequence order no longer encodes the geometry of the sample---so a positional mechanism indexed by sequence position can learn the memory layout rather than the object. This is a general failure mode for attention over packed geometric data. We introduce PACK3D, a local-global attention operator that separates memory layout from attention geometry. PACK3D keeps local neighborhood attention exact, compresses only distant context into contiguous 3D block summaries, and indexes every query-key interaction with axial RoPE on true token-grid coordinates rather than packed positions. Because the rotation is applied before a block is summarized, the query-to-summary logit remains a weighted sum of query-relative 3D displacements; compressed global context therefore stays spatially interpretable after packing, instead of collapsing to anonymous pooled content. We measure the failure mode directly. On a controlled synthetic task---train on one packing, change only the memory order of the tokens, hold the physical labeling rule fixed---learned positional embeddings and packed-sequence RoPE solve the training layout perfectly yet collapse under Morton, block-contiguous, and random repacking, while axial grid-coordinate RoPE stays exact throughout. The same coordinate system carries to real data: on Allen WTC-11 hiPSC microscopy, PACK3D improves over a parameter-matched 3D U-Net on the 15-structure joint task and over a matched learned-position local-global baseline, with controls isolating the coordinate system, not encoder capacity, as the source of the gain. A fused Triton implementation matches windowed-attention latency at 65k tokens while supplying the long-range context windowing cannot---so the operator is not only correct but trainable at scale. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.38694 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T06:07:08.671Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).