Metraipolis › archive › dc_b213ddab84da
Why Adaptive Optimizers Underestimate Rare Tokens — arXiv:2609.37535
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.37535
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T17:31:35.096Z
- SHA-256 of the archived text
7bf147014129f2e92f3a576fc0f043231ec702fca6f59927643cd493fb158410- Stable URL of this page
https://metraipolis.com/documento/dc_b213ddab84da
How to cite
Why Adaptive Optimizers Underestimate Rare Tokens — arXiv:2609.37535 — arXiv (en), https://arxiv.org/abs/2609.37535 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b213ddab84da (Metraipolis archive dc_b213ddab84da, sha256 7bf147014129f2e92f3a576fc0f043231ec702fca6f59927643cd493fb158410)
BibTeX
@misc{dcb213ddab84da,
title = {Why Adaptive Optimizers Underestimate Rare Tokens — arXiv:2609.37535},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.37535},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b213ddab84da; sha256 7bf147014129f2e92f3a576fc0f043231ec702fca6f59927643cd493fb158410},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37535. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
In the softmax output layer, a rare token receives a small positive logit gradient on most steps and a much larger negative gradient on the few steps when it is the target. SGD simply adds these contributions. Coordinate-wise adaptive methods such as Adam, RMSProp, and sign descent instead divide each update by a running estimate of its magnitude, and that estimate is largest immediately after the token appears. This imbalance has two effects. At the level of the whole output layer, we characterize which optimizers preserve the mean output embedding: every method whose update is linear in past gradients does, as do Kronecker-factored and orthogonalized methods such as Shampoo and Muon. Adam, Adafactor, Lion, and sign descent do not, and for these methods we obtain an exact step-by-step expression for the change. At the level of an individual rare token, the same normalization shifts the training fixed point. In the unigram model, sign descent lowers the logit of every token that occurs in fewer than half of the minibatches at a constant expected rate. For RMSProp with periodic arrivals, we can solve the fixed point in closed form: if a token is absent for at least two consecutive minibatches, its equilibrium probability is strictly below its data frequency for every learning rate, and the ratio tends to $\kappa/(2(e^{\kappa/2}-1))$. Here $\kappa$ is the mean number of steps between occurrences divided by the second-moment time constant $1/(1-\beta_2)$. In the same model, SGD and AMSGrad retain the unbiased fixed point. We test these predictions both in a unigram model and in a small language model trained from a known generating distribution. With random arrivals, the bias is larger than the periodic formula predicts; in the language model, the optimizers with the biased fixed point also fit the generating distribution less well.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2609.37535 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T16:59:48.343Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).