Metraipolis › archive › dc_0ce5cde96eb0

Theoretical Analysis of Muon on Transformers for Next-Token Prediction — arXiv:2611.04465

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04465
License (as archived)
CC0-1.0
Archived
2026-11-04T21:51:37.336Z
SHA-256 of the archived text
afbf8dc89d735663a29c9e2c6f6021a441040abb022d743795755e39bcb420c3
Stable URL of this page
https://metraipolis.com/documento/dc_0ce5cde96eb0

How to cite

Theoretical Analysis of Muon on Transformers for Next-Token Prediction — arXiv:2611.04465 — arXiv (en), https://arxiv.org/abs/2611.04465 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0ce5cde96eb0 (Metraipolis archive dc_0ce5cde96eb0, sha256 afbf8dc89d735663a29c9e2c6f6021a441040abb022d743795755e39bcb420c3)
BibTeX
@misc{dc0ce5cde96eb0,
  title = {Theoretical Analysis of Muon on Transformers for Next-Token Prediction — arXiv:2611.04465},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04465},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0ce5cde96eb0; sha256 afbf8dc89d735663a29c9e2c6f6021a441040abb022d743795755e39bcb420c3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04465. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Muon has recently shown strong empirical performance in training Transformer-based large language models (LLMs), yet its optimization dynamics remain poorly understood from a theoretical perspective. In this paper, we analyze the training dynamics of Muon for next-token prediction (NTP) in Transformers. The training procedure consists of two phases: the preprocessing phase learns the output-value component, while the main phase learns the key-query component. For the preprocessing phase, we characterize the optimization trajectory exactly and show that the normalized spectral margin is optimal after each update. For the main phase, we establish nonasymptotic convergence rates for the training loss and the normalized margin gap under softmax attention. These results suggest that Muon's update rule can help models efficiently learn which tokens matter for prediction, offering a possible explanation for its strong performance in large language model training. The learned weight structure also yields a generalization guarantee for unseen contexts. Furthermore, we validate our theoretical insights with experiments on next-token prediction tasks.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.04465 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T21:38:49.713Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).