Metraipolis › archive › dc_e99af91f0258
Backward-State Policy Is Part of the Learning Algorithm — arXiv:2609.39813
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.39813
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T13:46:39.861Z
- SHA-256 of the archived text
acc0117412d5d6d90da58f0ceb5e6e3ebc66e32de1d1f08bd28e0570a8dbd759- Stable URL of this page
https://metraipolis.com/documento/dc_e99af91f0258
How to cite
Backward-State Policy Is Part of the Learning Algorithm — arXiv:2609.39813 — arXiv (en), https://arxiv.org/abs/2609.39813 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e99af91f0258 (Metraipolis archive dc_e99af91f0258, sha256 acc0117412d5d6d90da58f0ceb5e6e3ebc66e32de1d1f08bd28e0570a8dbd759)
BibTeX
@misc{dce99af91f0258,
title = {Backward-State Policy Is Part of the Learning Algorithm — arXiv:2609.39813},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.39813},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e99af91f0258; sha256 acc0117412d5d6d90da58f0ceb5e6e3ebc66e32de1d1f08bd28e0570a8dbd759},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39813. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Low-precision training rounds tensors that the backward pass reads again, often for several gradients; each use can read the forward's rounded value, the original, or a new random rounding. This backward-state policy looks like a memory and precision detail, settled by copy accuracy and final loss. We argue that it is part of the learning algorithm, and that neither check shows whether it is right. Copy accuracy does not decide the outcome: in three pairs of 390M runs with an emulated FP8 backward, training fails when attention's backward reuses the forward's rounded output and succeeds with a new rounding from the same distribution. Even the most accurate copy, the original itself, can be wrong by our reference: the gradient of the forward pass as it actually ran, with gradients passed through rounding unchanged. For example, a normalization output stored in low precision feeds two gradients: the gain's gradient needs the original, but the next layer's weight gradient needs the rounded value that layer multiplied. Final loss, the other check, does not rule out the error of reading the original for both: it persists in models trained with such a store, while planned loss comparisons stay within a margin fixed in advance. We therefore derive from this reference which value each use must read, or which substitute gives the same gradient on average with the forward held fixed, and check these per-use requirements on single operators, without training. In three tests using PyTorch and Transformer Engine, the requirements predicted beforehand whether reuse changes what the backward computes on average relative to an independent copy, and every prediction held. Backward-state policy is thus part of the learning algorithm: it should be specified and checked use by use, not settled by copy accuracy and final loss. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.39813 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T13:18:10.463Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).