Metraipolis › archive › dc_5dd4e0627332

Rethinking Normalization Placement for LLMs: Post-Norm under Curriculum Depth Growing — arXiv:2608.13156

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.13156
License (as archived)
CC0-1.0
Archived
2026-10-19T04:17:34.209Z
SHA-256 of the archived text
3d677dff5035fa4fcb17f44b3e315dcc7e8279e06e0afc43cd2e1e764456d173
Stable URL of this page
https://metraipolis.com/documento/dc_5dd4e0627332

How to cite

Rethinking Normalization Placement for LLMs: Post-Norm under Curriculum Depth Growing — arXiv:2608.13156 — arXiv (en), https://arxiv.org/abs/2608.13156 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5dd4e0627332 (Metraipolis archive dc_5dd4e0627332, sha256 3d677dff5035fa4fcb17f44b3e315dcc7e8279e06e0afc43cd2e1e764456d173)
BibTeX
@misc{dc5dd4e0627332,
  title = {Rethinking Normalization Placement for LLMs: Post-Norm under Curriculum Depth Growing — arXiv:2608.13156},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.13156},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5dd4e0627332; sha256 3d677dff5035fa4fcb17f44b3e315dcc7e8279e06e0afc43cd2e1e764456d173},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.13156. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Pre-norm is the standard normalization placement in modern Transformers because it facilitates joint optimization of full-depth models. We ask whether this preference persists when depth is introduced through a curriculum. In curriculum depth growth, each appended block receives the boundary representation produced by a trained prefix, making normalization placement relevant to forward conditioning. We therefore test whether placement and training curriculum interact. In a controlled distillation study with a Qwen3-8B teacher and a nine-layer student, pre-norm and post-norm are indistinguishable under joint training, differing by $0.0004$ validation CE, while post-norm improves over pre-norm by $0.0328$ under curriculum growth, an order of magnitude larger. A post-joint control matched by student active-layer tokens remains worse than post-grow, which rules out compute as the sole explanation. The ranking crosses over during the curriculum: post-norm takes the lead once blocks are appended. Single-block and freeze controls localize the ranking change to block appending rather than shallow-block quality or retraining. Boundary diagnostics associate post-norm with stable residual scales and pre-norm with structural-token scale drift; on a fixed batch, the final pre-grow block is also nearly identity-mapped. Together with the phase-wise crossover, these observations are consistent with boundary-scale conditioning after new blocks are appended. The results motivate treating normalization placement and training curriculum as coupled design choices in this distillation setting.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-14 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.13156 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T04:13:06.586Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).