Metraipolis › archive › dc_dc20004dc939

Higher-Dimensional Rotary Position Embedding — arXiv:2608.29715

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.29715
License (as archived): CC0-1.0
Archived: 2026-10-17T07:52:13.913Z
SHA-256 of the archived text: 386e9d11012412ad8424486199951362c09ff090b8c8a62a21eb5d0f27724791
Stable URL of this page: https://metraipolis.com/documento/dc_dc20004dc939

How to cite

Higher-Dimensional Rotary Position Embedding — arXiv:2608.29715 — arXiv (en), https://arxiv.org/abs/2608.29715 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dc20004dc939 (Metraipolis archive dc_dc20004dc939, sha256 386e9d11012412ad8424486199951362c09ff090b8c8a62a21eb5d0f27724791)

BibTeX

@misc{dcdc20004dc939,
  title = {Higher-Dimensional Rotary Position Embedding — arXiv:2608.29715},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.29715},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dc20004dc939; sha256 386e9d11012412ad8424486199951362c09ff090b8c8a62a21eb5d0f27724791},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.29715. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Transformers rely on position embedding mechanisms in long context modeling in most cases. Rotary Position Embedding (RoPE) embeds positional information with independent 2D rotations, forming relative position terms in self-attention. However, its pairwise, block-based, and decoupled structure limits deep mixing and robustness across channels. We propose HD-RoPE, which extends RoPE from independent 2D rotations to higher-dimensional rotations and introduces a Paley-I orthogonal basis to obtain balanced, isotropic, and dense phase mixing within each rotation subspace. This significantly enhances channel coupling and rotational degrees of freedom while maintaining orthogonal stability and the relative position closure property. Furthermore, HD-RoPE is easily optimized for engineering efficiency without introducing additional trainable parameters. We have conducted extensive evaluation results demonstrating that HD-RoPE achieves significant performance improvements over standard RoPE across various popular benchmarks and in both long and short contexts.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.29715 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T07:28:26.796Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).