Metraipolis › archive › dc_62e37eeb6475
Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization — arXiv:2610.12444
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.12444
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T06:23:47.618Z
- SHA-256 of the archived text
740e5eac528e768b3f4d8f7e806d402bb34553df700e55a6b933eb8d67beafd6- Stable URL of this page
https://metraipolis.com/documento/dc_62e37eeb6475
How to cite
Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization — arXiv:2610.12444 — arXiv (en), https://arxiv.org/abs/2610.12444 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_62e37eeb6475 (Metraipolis archive dc_62e37eeb6475, sha256 740e5eac528e768b3f4d8f7e806d402bb34553df700e55a6b933eb8d67beafd6)
BibTeX
@misc{dc62e37eeb6475,
title = {Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization — arXiv:2610.12444},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.12444},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_62e37eeb6475; sha256 740e5eac528e768b3f4d8f7e806d402bb34553df700e55a6b933eb8d67beafd6},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.12444. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Quantizing AdamW's optimizer states reduces persistent storage, but quantization errors propagate through the moment recurrences and perturb subsequent adaptive updates. We redesign 4-bit optimizer-state quantization for AdamW from the perspective of \emph{rounding space}: the coordinate in which a quantizer chooses between adjacent reconstruction levels. For the second moment, a local analysis of the quantization cell adjacent to zero shows that small mean state error need not imply small mean preconditioner error at the next step. A one-dimensional quadratic construction further shows qualitatively different optimization dynamics under state-space and preconditioner-space rounding. These results motivate Zero-Inclusive Preconditioner-space Stochastic Rounding (\textbf{ZIP-SR}), which retains zero in the second-moment codebook and computes stochastic-rounding probabilities in preconditioner space. As a complementary route, Zero-Excluding EDEN calibration (\textbf{ZE-EDEN}) uses a zero-excluding second-moment codebook and rescales the quantized second-moment block to mitigate the preconditioner distortion caused by the positive quantization floor. Both configurations use 4-bit NormalFloat (NF4) for the first moment, with targeted stochastic rounding of the LM-head first moment during the final 10\% of training. Across GPT- and Llama-style pretraining experiments ranging from \textbf{130M} to \textbf{2.7B} parameters, both methods reduce TorchAO 4-bit AdamW's mean validation-loss gap to 32-bit AdamW at every evaluated model size, with the largest reported gap reduction reaching \textbf{70\%}. In full-parameter supervised fine-tuning, both recipes achieve lower validation loss than TorchAO while remaining close to 32-bit AdamW on downstream tasks.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.12444 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T05:50:31.805Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).