Metraipolis › archive › dc_7b7cf151f4e6

Replaying the Same Gradients at a Different $\varepsilon$: A Registered Test of the Fixed-Trajectory Account of Adam's $\varepsilon$ for One Rare Row — arXiv:2610.38169

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.38169
License (as archived): CC0-1.0
Archived: 2026-11-01T02:23:25.561Z
SHA-256 of the archived text: 4c1344b18718ad7e6bb0cffdacf967c2d26c0cc4e76c1b65252dac5646dbb617
Stable URL of this page: https://metraipolis.com/documento/dc_7b7cf151f4e6

How to cite

Replaying the Same Gradients at a Different $\varepsilon$: A Registered Test of the Fixed-Trajectory Account of Adam's $\varepsilon$ for One Rare Row — arXiv:2610.38169 — arXiv (en), https://arxiv.org/abs/2610.38169 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7b7cf151f4e6 (Metraipolis archive dc_7b7cf151f4e6, sha256 4c1344b18718ad7e6bb0cffdacf967c2d26c0cc4e76c1b65252dac5646dbb617)

BibTeX

@misc{dc7b7cf151f4e6,
  title = {Replaying the Same Gradients at a Different $varepsilon$: A Registered Test of the Fixed-Trajectory Account of Adam's $varepsilon$ for One Rare Row — arXiv:2610.38169},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.38169},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7b7cf151f4e6; sha256 4c1344b18718ad7e6bb0cffdacf967c2d26c0cc4e76c1b65252dac5646dbb617},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.38169. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Adam divides every update by the square root of a running average of squared gradients plus a small constant, epsilon. For an output-embedding row whose token is rare, which we call the probe row, raising epsilon can be read as shrinking that row's steps by arithmetic alone. We test that reading directly. While a training run proceeds, we also divide the probe row's update by a different epsilon, using the run's own moment estimates; this replay is exact arithmetic on the gradients the run actually saw, so it holds the run's trajectory fixed. A rule for when the replay counts as wrong was written down before the runs. We train a small causal transformer in a standard configuration. When epsilon rises by four orders of magnitude, the replay counts as wrong on each of the three seeds registered for the test: the probe row keeps 8.9--11.4 times more of its displacement than predicted with the main, sampled-negative head, and 5.0--7.7 times more with a full-softmax head scored as a second instrument. At three orders of magnitude, tested with the main head only, it does not, by narrow margins; a further two seeds, registered after this result had been committed and scored on their own, follow the same pattern. In separate real runs, raising epsilon on the probe row alone is enough to reproduce the row's change and raising it on every other parameter alone also moves the row part of the way --- readings about the runs that do not rescue the replay; scaling the row's learning rate instead, with the denominator untouched, is mispredicted by the replay in the same direction. This is one embedding row of one model family on one corpus; we make no claim about rates, about rows in general (other rows were read only as a panel on one trajectory) or about how the effect changes with model size.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.38169 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-01T02:12:44.105Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).