Metraipolis › archive › dc_dbd6284db339

Disentangling the Expressivity of RoPE — arXiv:2608.11909

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.11909
License (as archived): CC0-1.0
Archived: 2026-10-19T05:39:59.654Z
SHA-256 of the archived text: 8775d3c17863c6b6755931e5d135a70978089829c4a759673b61c22c99efea5d
Stable URL of this page: https://metraipolis.com/documento/dc_dbd6284db339

How to cite

Disentangling the Expressivity of RoPE — arXiv:2608.11909 — arXiv (en), https://arxiv.org/abs/2608.11909 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dbd6284db339 (Metraipolis archive dc_dbd6284db339, sha256 8775d3c17863c6b6755931e5d135a70978089829c4a759673b61c22c99efea5d)

BibTeX

@misc{dcdbd6284db339,
  title = {Disentangling the Expressivity of RoPE — arXiv:2608.11909},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.11909},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dbd6284db339; sha256 8775d3c17863c6b6755931e5d135a70978089829c4a759673b61c22c99efea5d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.11909. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Two accounts recur in explanations of the success of rotary position embeddings (RoPE). Expressivity studies associate periodic position information with modular predicates, whereas mechanistic and long-context studies emphasize positional anchors and local offsets. We formalize both accounts for fully uniform, finite-precision soft-attention transformers. We find that, if every rotary component is periodic, RoPE transformers recognize exactly the languages definable in past temporal logic with modular predicates. Conventional RoPE is different: The rotations it computes never repeat. This yields a precision-dependent bounded simulation of fixed-offset look-back operators, rather than an all-length modular characterization. Controlled experiments match this separation: Constructed periodic schedules length-generalize on modular languages, while conventional RoPE behaves more like a bounded locality bias and can impair tasks requiring position-invariant access to distant context. Altogether, our findings shed light on RoPE transformers, bringing theoretical expressivity characterizations closer to models used in practice.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-13 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Formal Languages and Automata Theory (cs.FL).
Pagina del lavoro: https://arxiv.org/abs/2608.11909 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:18:32.496Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).