Metraipolis › archive › dc_989bb2b9f082

Tune the Baseline First: Most of Muon's Scaled Weight Decay Gain Is a Tuning Gap, but Its Small Real Advantage Grows With Model Size — arXiv:2611.02529

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.02529
License (as archived)
CC0-1.0
Archived
2026-11-03T18:23:41.913Z
SHA-256 of the archived text
b0af93060b307eea7dab971e709f989dd6ef92ed82e3a72545f12db2ea736167
Stable URL of this page
https://metraipolis.com/documento/dc_989bb2b9f082

How to cite

Tune the Baseline First: Most of Muon's Scaled Weight Decay Gain Is a Tuning Gap, but Its Small Real Advantage Grows With Model Size — arXiv:2611.02529 — arXiv (en), https://arxiv.org/abs/2611.02529 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_989bb2b9f082 (Metraipolis archive dc_989bb2b9f082, sha256 b0af93060b307eea7dab971e709f989dd6ef92ed82e3a72545f12db2ea736167)
BibTeX
@misc{dc989bb2b9f082,
  title = {Tune the Baseline First: Most of Muon's Scaled Weight Decay Gain Is a Tuning Gap, but Its Small Real Advantage Grows With Model Size — arXiv:2611.02529},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02529},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_989bb2b9f082; sha256 b0af93060b307eea7dab971e709f989dd6ef92ed82e3a72545f12db2ea736167},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02529. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Scaled weight decay for Muon (Muon-SW) has been reported to train language models more efficiently than constant decoupled weight decay, but such comparisons usually reuse one learning-rate (LR) and weight-decay (WD) setting for both methods. We re-run the comparison under the MAI-Base-2 protocol: LR and WD are tuned jointly through their product for each method, on a source-scaled mixture-of-experts (MoE) scaling ladder (1.9M to 15M active parameters) that holds tokens per active parameter and data repetition fixed across sizes. We find: (1) Most of the gain is a tuning gap. At the shared setting Muon-SW wins by 0.035 bits per byte (BPB), a 1.52x compute saving; after both methods are tuned, the gap is 0.0026 BPB (95% CI [0.0018, 0.0034]), a 1.03x saving, so about 90% of the advantage disappears. (2) The small advantage that remains is real: at the smallest size, all four paired seeds favour Muon-SW at the tuned settings. (3) That advantage grows with model size: across an 8x range of active parameters, the tuned gap rises from 0.0036 to 0.0051 BPB while the untuned gap shrinks from 0.035 to 0.024, so the surviving share rises from about 10% to 21%. (4) The best weight decay moves with scale: the optimal LRWD product grows for both methods, faster for Muon-SW (ratio 1.40 to 1.91), and a law fitted on the two smallest sizes overshoots the third by 1.3-1.5x, costing up to 0.002 BPB. The whole study runs on one NVIDIA T4 GPU; we release the code and all run results.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.02529 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:10:07.971Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).