Metraipolis › archive › dc_6e330a12ecd3
The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics — arXiv:2609.35392
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.35392
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:03:33.427Z
- SHA-256 of the archived text
05a239587d70981a240420a2584a37c1650c2e19c5d7053d1d4c3ce03426f5c3- Stable URL of this page
https://metraipolis.com/documento/dc_6e330a12ecd3
How to cite
The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics — arXiv:2609.35392 — arXiv (en), https://arxiv.org/abs/2609.35392 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6e330a12ecd3 (Metraipolis archive dc_6e330a12ecd3, sha256 05a239587d70981a240420a2584a37c1650c2e19c5d7053d1d4c3ce03426f5c3)
BibTeX
@misc{dc6e330a12ecd3,
title = {The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics — arXiv:2609.35392},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.35392},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6e330a12ecd3; sha256 05a239587d70981a240420a2584a37c1650c2e19c5d7053d1d4c3ce03426f5c3},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35392. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Adam is the default optimizer for training modern deep neural networks, yet its adaptive behavior remains poorly understood due to the complex interaction between its first- and second-moment exponential moving averages (EMAs). We study Adam in the tied-$\beta$ regime, where the two EMA decay rates are equal, and show that its adaptive dynamics can be expressed through a transformed ratio with approximately scale-stable behavior. Empirically, this transformed ratio exhibits a stable, heavy-tailed distribution across tasks, model scales, and training stages, in contrast to the variability of raw moment magnitudes. This empirical stability has both practical and conceptual consequences. First, we derive a recurrence for the transformed ratio, yielding a reparameterization of Adam that replaces the second moment with a compressible state. Leveraging its stable distribution, we show that a fixed 4-bit codebook is sufficient in our experiments to store this state without auxiliary scaling, achieving performance competitive with full-precision Adam. Second, the transformed ratio view clarifies Adam's connection to sign-based methods: Adam reduces to sign-based momentum modulated by the transformed ratio, and replacing it with a constant recovers Signum as a limiting case. This perspective further provides a simple rule for transferring learning rates between the two methods. Together, these results suggest that tied-$\beta$ Adam admits a simple and approximately stable ratio structure underlying its adaptive behavior and demonstrate its utility for both analysis and efficient implementation. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.35392 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:03.345Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).