Metraipolis › archive › dc_df9cf08933fb

Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule — arXiv:2608.19491

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.19491
License (as archived)
CC0-1.0
Archived
2026-10-18T11:39:26.975Z
SHA-256 of the archived text
60bbf2ed01dd029d248ae94ab8359d67262c21b89c3bff21b8982cf960de2e3b
Stable URL of this page
https://metraipolis.com/documento/dc_df9cf08933fb

How to cite

Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule — arXiv:2608.19491 — arXiv (en), https://arxiv.org/abs/2608.19491 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_df9cf08933fb (Metraipolis archive dc_df9cf08933fb, sha256 60bbf2ed01dd029d248ae94ab8359d67262c21b89c3bff21b8982cf960de2e3b)
BibTeX
@misc{dcdf9cf08933fb,
  title = {Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule — arXiv:2608.19491},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.19491},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_df9cf08933fb; sha256 60bbf2ed01dd029d248ae94ab8359d67262c21b89c3bff21b8982cf960de2e3b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.19491. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Most modern optimizers form their momentum as an exponential moving average (EMA) of past gradients, forgetting every direction at one fixed rate. However, the inputs a deep network sees during training can be highly anisotropic, with a few directions queried frequently while most are seen rarely. Preconditioning methods address this anisotropy by wrapping extra processing around this buffer and leave the momentum update itself unchanged. We propose Activation-Keyed Momentum (AK-Momentum), which builds direction-awareness into the momentum update rule. The gradient of a linear layer splits into an input activation that acts as a key and an output-side error that acts as a value. Keying on that activation, AK-Momentum updates the momentum buffer by the canonical delta rule, so each direction is forgotten at a rate set by how often it appears. We prove that it is a valid momentum, that it applies the input-side curvature correction without matrix inversion, and that it clears stale directions faster than EMA under both a fixed and a drifting optimum. It is a drop-in replacement for the momentum buffer of any optimizer, its coefficient transfers across widths under $\mu$P, and its extra compute stays between $22.2\%$ and $25.0\%$ of a gated-MLP block's linear cost with no persistent memory. In FineWeb-Edu pretraining, AdamW with AK-Momentum (AK-AdamW) reaches AdamW's validation loss in up to $46.39 \pm 4.32\%$ fewer steps at 67M and $22.12 \pm 0.80\%$ at 370M over three seeds, and the gain persists at 1B on a Chinchilla-optimal budget. A Muon baseline tuned under the same protocol sits above AK-AdamW at both language-model scales, and the gain holds for SGD, ResNet-18, and ViT-Tiny on CIFAR-10. Training-time diagnostics confirm the predicted mechanism, better gradient tracking and healthier input directions.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-21 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL); Optimization and Control (math.OC); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2608.19491 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T11:38:27.941Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).