Metraipolis › archive › dc_be22aee46518

bf16 Rounding Silently Discards Most RL Policy Updates: A Hidden Training-Inference Mismatch and an Unbiased One-Line Fix — arXiv:2610.41073

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.41073
License (as archived): CC0-1.0
Archived: 2026-11-02T04:44:34.967Z
SHA-256 of the archived text: 41a49261ea64b44cd5d2830a2ff589330760450a9667872e8db0e58ea7435c80
Stable URL of this page: https://metraipolis.com/documento/dc_be22aee46518

How to cite

bf16 Rounding Silently Discards Most RL Policy Updates: A Hidden Training-Inference Mismatch and an Unbiased One-Line Fix — arXiv:2610.41073 — arXiv (en), https://arxiv.org/abs/2610.41073 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_be22aee46518 (Metraipolis archive dc_be22aee46518, sha256 41a49261ea64b44cd5d2830a2ff589330760450a9667872e8db0e58ea7435c80)

BibTeX

@misc{dcbe22aee46518,
  title = {bf16 Rounding Silently Discards Most RL Policy Updates: A Hidden Training-Inference Mismatch and an Unbiased One-Line Fix — arXiv:2610.41073},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41073},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_be22aee46518; sha256 41a49261ea64b44cd5d2830a2ff589330760450a9667872e8db0e58ea7435c80},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41073. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning (RL) post-training of language models keeps fp32 master weights in the optimizer and serves rollouts from a bf16 copy produced by round-to-nearest conversion. At the learning rates in common use, most per-step updates are smaller than half a bf16 spacing and the conversion discards them: on Qwen3-1.7B only 0.4-0.5% of served weights change per step, and the policy that generates the training data lags the optimizer by 50-150 steps. Existing mismatch corrections cannot see this lag because they compare two copies downstream of the same rounding. We propose Fair Cast: export the master with an unbiased rounding rule (stochastic rounding or error feedback) and compute the training loss on the served copy. The change is one line at the export call, adds no hyperparameters and costs 22-38 ms per export of 1.7B parameters. Across 25 matched seed-pairs on Qwen3-0.6B to 4B with AdamW and Muon, Fair Cast is ahead on 18, tied on 6 and behind on 1, with a mean gain of 6.1 points; on zebra puzzles it reaches 0.57 +/- 0.16 over four runs where round-to-nearest collapses to 0.33 +/- 0.10. It also makes the saved fp32 checkpoint agree with the served model, and it exposes the implicit lag of round-to-nearest as a regularizer that can be restored explicitly when a task needs it.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.41073 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T04:42:12.445Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).