Metraipolis › archive › dc_fe628f81934f
What a Weight Update Buys: Leverage Across Training Stages — arXiv:2610.37719
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.37719
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T06:17:13.676Z
- SHA-256 of the archived text
c01824fe64e11fd753733fb9b227ae912cd946608ff52e5e5d65e909acd6dcb2- Stable URL of this page
https://metraipolis.com/documento/dc_fe628f81934f
How to cite
What a Weight Update Buys: Leverage Across Training Stages — arXiv:2610.37719 — arXiv (en), https://arxiv.org/abs/2610.37719 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fe628f81934f (Metraipolis archive dc_fe628f81934f, sha256 c01824fe64e11fd753733fb9b227ae912cd946608ff52e5e5d65e909acd6dcb2)
BibTeX
@misc{dcfe628f81934f,
title = {What a Weight Update Buys: Leverage Across Training Stages — arXiv:2610.37719},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.37719},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fe628f81934f; sha256 c01824fe64e11fd753733fb9b227ae912cd946608ff52e5e5d65e909acd6dcb2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.37719. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Post-training moves a language model's weights by about a thousandth of their norm and changes its behavior sharply; pretraining moves them by a third and buys a slow drift. We introduce leverage, an exact measure of what a unit of weight change buys: the Fisher quadratic form of an update, computed with one forward-mode pass per direction, divided by the same form for random directions of the same size at the same checkpoint. Leverage orders released Olmo-3, Tülu-3, Zephyr, and OLMo-2 checkpoints over four orders of magnitude: pretraining windows sit at the random level on prose, SFT five times above it, RL-Zero and RLVR at 400-1,500, and DPO at 400-23,000. With this instrument we find that pretraining accumulates into noise: windows of 30k to 1M steps are no more consequential on prose than random directions of their size, as a noisy quadratic model of SGD predicts. We find that training precision changes what an update looks like: training bf16 parameters without an fp32 master copy freezes 36-99% of coordinates, lowers leverage up to sevenfold, and creates an alignment between update magnitudes and the diagonal Fisher that fp32 training does not show. We derive and verify a law for reuse: the sign-flip null predicts the relative damage of random drop-and-rescale (DARE) at keep rate $p$ as $(1-p)/(p\,\rho_{\mathrm{sign}})$, within a factor of two of measurement wherever DARE succeeds. Two controls make these readings reliable: every controlled SFT and DPO trajectory with fp32 master weights loses leverage as it moves, and two re-runs of a released DPO stage from its anchor with its published recipe land within 1% of the released displacement and 4% of its leverage, where a plain recipe reads five to six times lower; leaving one component out traces the level to the length-normalized loss and its batch size of 128 together.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.37719 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T06:07:23.386Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).