Metraipolis › archive › dc_029a6daee92f

LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition — arXiv:2610.09179

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.09179
License (as archived)
CC0-1.0
Archived
2026-10-08T11:10:38.967Z
SHA-256 of the archived text
44795f5892d9acf221970e73105e046f4b744908c7b66928b80c8912f47c1652
Stable URL of this page
https://metraipolis.com/documento/dc_029a6daee92f

How to cite

LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition — arXiv:2610.09179 — arXiv (en), https://arxiv.org/abs/2610.09179 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_029a6daee92f (Metraipolis archive dc_029a6daee92f, sha256 44795f5892d9acf221970e73105e046f4b744908c7b66928b80c8912f47c1652)
BibTeX
@misc{dc029a6daee92f,
  title = {LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition — arXiv:2610.09179},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.09179},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_029a6daee92f; sha256 44795f5892d9acf221970e73105e046f4b744908c7b66928b80c8912f47c1652},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09179. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As data propagates through a Transformer, the norm of its hidden states grows by orders of magnitude with depth, a phenomenon framed as 'curse of depth' and nearly universally treated as a pathology to be suppressed. We take the opposite view. Across 16 pre-trained LLMs from 9 families, spanning dense, mixture-of-experts and hybrid architectures and Pre-, Peri- and Post-Norm designs, we find that this growth reflects an emergent depth-positional encoding, carried by the only learned per-layer gain on the residual stream, the normalization weight $\gamma$: with depth, $\gamma$ grows in magnitude and rotates in direction, jointly encoding the layer index. We make this depth-conditioned encoding explicit with LayerRoPE, an implicit analog of RoPE along the depth axis, which replaces all layerwise $\gamma$ vectors with a single shared vector and depth-conditioned scalars, at a net reduction in parameters and $<0.02\%$ change in FLOPs. Across a model ladder scaled up to $100$B+ tokens, LayerRoPE consistently outperforms Pre-, Post- and Peri-Norm and Layer-Norm Scaling, reaching Pre-Norm's 1.3B loss with $3.4\times$ less compute; LayerRoPE is the only approach that shows strong convergence and improves near monotonically as depth scales to 512 layers. It improves learning-rate sensitivity by $3$-$10\times$, and transfers naively to and consistently improves looped latent models and Vision Transformers. Inspecting its learned schedule inverts the prevailing premise: LayerRoPE does not shrink the residual stream but widens it, damping what each block reads while amplifying what it writes. Depth stability, our results suggest, calls not for suppressing the residual stream, but for depth-conditioned regulation of the computational blocks it feeds.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.09179 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:14:06.055Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).