Metraipolis › archive › dc_3cfa7a70d4db
Lost in the bf16 Cast: Exporting Ternary Language Models Can Revert Most Low-Learning-Rate Code Changes — arXiv:2610.07853
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.07853
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T11:23:45.245Z
- SHA-256 of the archived text
fd3ce6517b5e78c7fc950f5f30d68d112edd8898382db874395a1ab75424c261- Stable URL of this page
https://metraipolis.com/documento/dc_3cfa7a70d4db
How to cite
Lost in the bf16 Cast: Exporting Ternary Language Models Can Revert Most Low-Learning-Rate Code Changes — arXiv:2610.07853 — arXiv (en), https://arxiv.org/abs/2610.07853 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3cfa7a70d4db (Metraipolis archive dc_3cfa7a70d4db, sha256 fd3ce6517b5e78c7fc950f5f30d68d112edd8898382db874395a1ab75424c261)
BibTeX
@misc{dc3cfa7a70d4db,
title = {Lost in the bf16 Cast: Exporting Ternary Language Models Can Revert Most Low-Learning-Rate Code Changes — arXiv:2610.07853},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.07853},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3cfa7a70d4db; sha256 fd3ce6517b5e78c7fc950f5f30d68d112edd8898382db874395a1ab75424c261},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07853. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Ternary language models such as BitNet b1.58, Falcon-E and BitCPM are fine-tuned with higher-precision latent weights and deployed as ternary codes produced by an export step that, in the labs' documented pipelines, first casts the latents to bf16. We audit those pipelines across three labs. In released checkpoints, fp32 quantization of the shipped latents disagrees with the deployed codes on 0.83-1.77% of codes in Falcon-E and BitCPM and on 1.530% in BitNet 2B-4T; for Falcon-E and BitCPM most disagreements are products that bf16 rounding lands exactly on the threshold, which ties-to-even maps to zero, and the unmodified onebitllms exporter reproduces all four Falcon-E releases byte for byte. At fine-tuned endpoints, with learning rates selected to match a nominal learning-rate-to-bf16-ULP ratio, the documented export lowers greedy GSM8K strict accuracy from 58.79% to 0.78% for Falcon-E-1B-Base and from 36.13% to 0.39% for BitCPM-CANN-0.5B, and a bf16 save and reload lowers BitNet 2B-4T's strict accuracy by 27.54 points while its last-number accuracy rises. Two compatibility remedies, writing the training quantizer's codes directly or adjusting the bf16 inputs until the unchanged tools emit them, each met a 4-point strict-accuracy non-inferiority criterion against online evaluation in all three models. In two model families, randomized interventions on the initial distance from the threshold support distance-dependent selection of the codes that fine-tuning changes. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.07853 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T11:15:39.491Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).