Metraipolis › archive › dc_d2cccf7b17bc

Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training — arXiv:2609.36692

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.36692
License (as archived)
CC0-1.0
Archived
2026-10-07T16:26:17.819Z
SHA-256 of the archived text
9e0d949c892c8f63ba660557406bb1e52583803324d05d37dcd64c3f20d6d11f
Stable URL of this page
https://metraipolis.com/documento/dc_d2cccf7b17bc

How to cite

Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training — arXiv:2609.36692 — arXiv (en), https://arxiv.org/abs/2609.36692 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d2cccf7b17bc (Metraipolis archive dc_d2cccf7b17bc, sha256 9e0d949c892c8f63ba660557406bb1e52583803324d05d37dcd64c3f20d6d11f)
BibTeX
@misc{dcd2cccf7b17bc,
  title = {Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training — arXiv:2609.36692},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.36692},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d2cccf7b17bc; sha256 9e0d949c892c8f63ba660557406bb1e52583803324d05d37dcd64c3f20d6d11f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36692. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Matrix optimizers have emerged as a promising direction, with Muon standing out as a prominent design. Revisiting Muon through its full-Gram representation, we observe that it jointly processes marginal-scale and interaction information. This opens an alternative way to organize geometric information hierarchically, motivating the Normalize-Then-Precondition framework. Specifically, it first uses diagonal-Gram information to construct a marginally normalized update, then applies spectral preconditioning to its directional interaction geometry. Building on this framework, we develop NormPre with NormPre-G and NormPre-L adopting global and localized spectral preconditioning, grounded in spectral-norm steepest descent and a regularized formulation followed by leading mode selection, respectively. To enable large-scale training, NormPre-G uses Newton-Schulz iterations and NormPre-L employs randomized sketching to approximate the leading interaction eigenspace. Theoretically, we establish $\mathcal{O}(T^{-1/2})$ convergence guarantees for simplified versions of NormPre. Across extensive pretraining experiments on GPT-2 Small, LLaMA and Qwen3, both variants consistently outperform AdamW, Muon and MANO under matched training budgets. Further efficiency and spectral analyses reveal the complementary strengths of two variants and characterize their performance-efficiency trade-off. We open-source our code through a GitHub repository at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.36692 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:53.882Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).