Metraipolis › archive › dc_7bc4587a6fb6

WavePrune: One period is often enough for RoPE — arXiv:2610.06963

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.06963
License (as archived)
CC0-1.0
Archived
2026-10-07T11:22:58.650Z
SHA-256 of the archived text
d607ebabbf52b1bc3d840b274d49e4ea42d39e22f37a7bf8b123459564ad5d2d
Stable URL of this page
https://metraipolis.com/documento/dc_7bc4587a6fb6

How to cite

WavePrune: One period is often enough for RoPE — arXiv:2610.06963 — arXiv (en), https://arxiv.org/abs/2610.06963 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7bc4587a6fb6 (Metraipolis archive dc_7bc4587a6fb6, sha256 d607ebabbf52b1bc3d840b274d49e4ea42d39e22f37a7bf8b123459564ad5d2d)
BibTeX
@misc{dc7bc4587a6fb6,
  title = {WavePrune: One period is often enough for RoPE — arXiv:2610.06963},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.06963},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7bc4587a6fb6; sha256 d607ebabbf52b1bc3d840b274d49e4ea42d39e22f37a7bf8b123459564ad5d2d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.06963. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Rotary Position Embedding (RoPE) encodes token positions by rotating each two-dimensional channel of the query and key vectors at a channel-specific frequency, making the attention logits invariant to a common shift of positions. However, this rotation is periodic, and it leads to position aliasing where relative positions separated by a full rotation period become hard to tell apart. To address this, we propose WavePrune, which restricts each channel to its first rotation period. We show that it removes the distractions in attention maps created by position aliasing and improves overall long-context performance. Specifically, WavePrune raises the HELMET score on four of five models we test without any extra tuning (e.g., 35.7 -> 40.0 on Qwen3-8B). When pretraining models from scratch, WavePrune also achieves lower validation loss at extrapolated lengths than pretraining without it. Because WavePrune restricts each channel to a sliding window, it induces a fine-grained sparsity that our hardware-aligned CUDA kernels exploit for 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context. Together, these results show that RoPE's periodic structure, widely regarded as essential, is largely redundant beyond the first rotation period.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Sound (cs.SD).
Pagina del lavoro: https://arxiv.org/abs/2610.06963 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:15:39.408Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).