Metraipolis › archive › dc_514b6b384e8f

RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings — arXiv:2609.34556

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.34556
License (as archived): CC0-1.0
Archived: 2026-10-07T21:53:28.091Z
SHA-256 of the archived text: 661881e8f1e5e3544dc192785d16103740862a95f1045b57eaaee2c407951679
Stable URL of this page: https://metraipolis.com/documento/dc_514b6b384e8f

How to cite

RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings — arXiv:2609.34556 — arXiv (en), https://arxiv.org/abs/2609.34556 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_514b6b384e8f (Metraipolis archive dc_514b6b384e8f, sha256 661881e8f1e5e3544dc192785d16103740862a95f1045b57eaaee2c407951679)

BibTeX

@misc{dc514b6b384e8f,
  title = {RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings — arXiv:2609.34556},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.34556},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_514b6b384e8f; sha256 661881e8f1e5e3544dc192785d16103740862a95f1045b57eaaee2c407951679},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34556. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Transformers process tokens without any inherent notion of order, making positional encoding a fundamental requirement rather than an architectural refinement. Rotary Position Embedding (RoPE) has become the default positional encoding in modern language models, yet it is heavily biased toward nearby tokens. Existing alternatives have been evaluated under different settings, leaving the literature fragmented and without a clear replacement. We bring structure to this landscape by examining a specific weakness of RoPE: its slow frequency bands, whose wavelengths exceed the training context and expose models to unseen angles during extrapolation. We therefore introduce Data aware RoPE (DaRoPE), which preserves standard RoPE on the fast bands but replaces absolute position on the slow bands with bounded coordinates learned from contextual representations. Therefore, the slow-band geometry depends on the data rather than only on positional distance. We compare representative encodings under matched conditions across synthetic tasks, symbolic music, genomics, neural signals, and language models spanning 124M to 50B parameters. Across these experiments, DaRoPE leads on non-text benchmarks, mitigates recency bias, while remaining best or on par in language modeling and length extrapolation. Moreover, the learned coordinates also make the mechanism interpretable, revealing how attention layers leverage contextual information beyond token distance. Together, these results support DaRoPE as the best overall default among the evaluated methods, when there is no domain-specific reasons to prefer another.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.34556 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T21:27:08.624Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).